You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
In the path to have more stable metrics in our synthetic testing we have been trying out mahimahi and WebPageReplay to record and replay Wikipedia. For mahimahi we have used patched version fixed by Gilles over Benedikt Wolters HTTP2 version of https://github.com/worenga/mahimahi-h2o. With WebPageReplay we use the default version. The work has been done in T176361.
We have put mahimahi on ice because it is too much of hack and WebPageReplay is more a finished product. However right is our main concern that we only can use Chrome, we should make sure we could also run with Firefox.
The current version run that collect the data for https://grafana.wikimedia.org/dashboard/db/webpagereplay is a Docker container with this setup:
https://github.com/soulgalore/browsertime-replays/tree/master/webpagereplay and the setup looks like this:
Running on AWS (instance type c4.large) we get stable metrics. On desktop we can use 30 frames per second for the video and then get a change of 33 ms for first visual change. Speed Index got more variance but still ok (less than 50 points but it depends on the content). For emulated mobile we can run 60 frames per second and get the same first visual change and Speed Index somewhere of lower 50 points. We run the both desktop and mobile on 100 ms latency.
WebPageTest vs WebPageReplay
Let us compare what the metrics looks like comparing WebPageTest vs WebPageReplay.