You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
The AS Report (Autonomous Systems performance report) is a monthly-generated dataset anonymised from sampled visits to Wikipedia and other Wikimedia Foundation sites. The ranking is generated based on normalized RUM navtiming metrics measured in the web browser from sampled page views.
Performance metrics are collected using the Navigation Timing API.
Our CPU microbenchmark (source code) is run for a short time in a separate Worker thread, to avoid disrupting their experience. The benchmark assesses the overall performance of the device at the time of the measurement. This is important, as background memory usage and battery levels can greatly influence the performance of mobile devicess at any given time. In the wild, we have observed wide ranges of CPU scores for the exact same device model.
Using this information, we only keep the client metrics for a range of CPU scores. We then compile medians for that device range for each autonomous system, based on the visitors' IP addresses (source code).
Mobile vs desktop
We separate mobile and desktop experiences, as they have significantly different median page weights. As such, scores aren't comparable between mobile and desktop for a given country. Just as they are not comparable between countries either, due to different CPU benchmark score slices being selected for each. Reports are per country, as Internet services tend to be sold to consumers on a national basis, and networks in the same country face the same infrastructural challenges.
Mobile networks are those with at least one sampled visit having a "cellular" connection type, as reported by the Network Information API. When calculating the scores for mobile, we only consider data from sampled pageviews to the mobile site. Desktop networks are those with at least one sampled visit having a "wifi" connection type. When calculating scores for desktop, we only consider data from desktop pageviews.
We don't want an AS corresponding to an ISP selling femtocell devices widely to be unfairly advantaged or disadvantaged due to differences between the mobile and desktop experience, resulting in lighter or heavier pages on average.
Only AS organizations for which we found sufficient data are retained. Finally, only countries with sufficient data from sampled page views are reported.
We report the medians for 2 core RUM metrics measures by the visitors' browsers: Time to first byte and Page Load Time, collected using the Navigation Timing API.
Time to first byte (TTFB) is how long it takes between the client requesting the page and it receiving the first byte of data from us. Page load time (PLT) is how long it takes to load the whole page, including all images and critical styles/scripts.
TTFB is the metric closest to latency, which is something ASOs might improve by peering with us or tweaking their routes to us. PLT is the metric that correlates the most to the visitors' perception of performance, as shown by research we've conducted. It's what captures the ASOs' quality of service to their customers the best.
We also report the median transferSize as reported by the Navigation Timing API as a sanity check, to ensure that the RUM metrics comparison is fair between ASOs, and that the differences aren't caused by visitors using a particular ASO accessing much smaller or much bigger pages on average.
In order to respect the privacy of our visitors, we only report ASOs for which we aggregate more than 500 unique pageviews to generate scores.
This script is meant to be run on a stat machine on cron on the 1st of every month, using the following syntax:
python asoranking.py --publish
Which will generate the ranking for the previous calendar month in the form of a tsv file published to /
This is in turn picked up periodically by the Performance Team when regenerating performance.wikimedia.org.
- Introducing: Autonomous Systems report (2019), by Gilles Dubuc, techblog.wikimedia.org.