I set up alerts for average response time greater then 15s for last 5min in new Azure Portal.
After set up I immediately got an alert email with text: 'Average Response Time GreaterThan 15 (Milliseconds) in the last 5 minutes' was activated for Website: xxxxxx (Default-Web-WestEurope).
In portal I see seconds but it measures milliseconds!
I tried to report this behavior via our paid Azure support but I always got an error...