You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Performance/WebPageReplay/Alerts

From Wikitech-static
< Performance‎ | WebPageReplay
Revision as of 12:59, 9 March 2018 by imported>Phedenskog (Right now only alerts for 100 ms)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

We use Grafana alerts and Icinga to get automated alerts when we have a performance regression.

Mobile

We test the mobile site using Chrome running in emulated mode.

Setup

At the moment we test three pages for mobile:

We do 7 runs against WebPageReplay per URL, record a video in 60 fps and take the median metrics. Since we run against a local proxy (WebPageReplay) we set the latency to 100ms to slowdown the rendering.

We collect Visual Metrics (SpeedIndex/First Visual Change) and CPU metrics (time spent in Layout/Scripting/Loading/Painting) that we get from parsing the Chrome trace log.

Alerts

Alerts for First Visual Change and Speed Index looks for a 2% regression on all three pages and then alert.

Alerts for Layout/Scripting/Loading/Painting looks for a regression in 20 ms on all three pages for one of the metrics.

Alert dashboard

https://grafana.wikimedia.org/dashboard/db/webpagereplay-mobile-alerts

Desktop

We use Chrome and Firefox to collect metrics for desktop.

Setup

At the moment we test three pages for desktop:

We do 11 runs against WebPageReplay per URL, record a video in 60 fps for Firefox and 30 fps with Chrome and take the median metrics. Since we run against a local proxy (WebPageReplay) we test with two different latencies 50ms and 100ms to slowdown the rendering.

We collect Visual Metrics (SpeedIndex/First Visual Change) and CPU metrics (time spent in Layout/Scripting/Loading/Painting Chrome only) that we get from parsing the Chrome trace log.

Alerts

We have a little different setup depending on browser. At the moment we only alert on 100 ms but let us add alerts for 50 ms too.

Chrome

Alerts for First Visual Change and Speed Index looks for a 2% regression on all three pages and then alert.

Alerts for Layout/Scripting/Loading/Painting looks for a regression in 20 ms on all three pages for one of the metrics.

Firefox

The current version of Firefox seems really sensitive to content changes so we have a little higher limit there for regressions. Alerts for First Visual Change and Speed Index looks for a 5% regression on all three pages and then alert.

Alert dashboard

https://grafana.wikimedia.org/dashboard/db/webpagereplay-desktop-alerts