You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Performance/AS Report

From Wikitech-static
< Performance
Revision as of 03:53, 30 June 2022 by imported>Krinkle
Jump to navigation Jump to search

The AS Report (Autonomous Systems performance report) is a monthly-generated dataset anonymised from sampled visits to Wikipedia and other Wikimedia Foundation sites. The ranking is generated based on normalized RUM navtiming metrics measured in the web browser from sampled page views.

Methodology

Performance metrics are collected using the Navigation Timing API.

CPU microbenchmark

Our CPU microbenchmark (source code) is run for a short time in a separate Worker thread, to avoid disrupting their experience. The benchmark assesses the overall performance of the device at the time of the measurement. This is important, as background memory usage and battery levels can greatly influence the performance of mobile devicess at any given time. In the wild, we have observed wide ranges of CPU scores for the exact same device model.

Using this information, we only keep the client metrics for a range of CPU scores. We then compile medians for that device range for each autonomous system, based on the visitors' IP addresses.

Mobile vs desktop

We separate mobile and desktop experiences, as they have significantly different median page weights. As such, scores aren't comparable between mobile and desktop for a given country. Just as they are not comparable between countries either, due to different CPU benchmark score slices being selected for each. Reports are per country, as Internet services tend to be sold to consumers on a national basis, and networks in the same country face the same infrastructural challenges.

Mobile networks are those with at least one sampled visit having a "cellular" connection type, as reported by the Network Information API. When calculating the scores for mobile, we only consider data from sampled pageviews to the mobile site. Desktop networks are those with at least one sampled visit having a "wifi" connection type. When calculating scores for desktop, we only consider data from desktop pageviews.

We don't want an AS corresponding to an ISP selling femtocell devices widely to be unfairly advantaged or disadvantaged due to differences between the mobile and desktop experience, resulting in lighter or heavier pages on average.

Only AS organizations for which we found sufficient data are retained. Finally, only countries with sufficient data from sampled page views are reported.

RUM metrics

We report the medians for 2 core RUM metrics measures by the visitors' browsers: Time to first byte and Page Load Time, collected using the Navigation Timing API.

Time to first byte (TTFB) is how long it takes between the client requesting the page and it receiving the first byte of data from us. Page load time (PLT) is how long it takes to load the whole page, including all images and critical styles/scripts.

TTFB is the metric closest to latency, which is something ASOs might improve by peering with us or tweaking their routes to us. PLT is the metric that correlates the most to the visitors' perception of performance, as shown by research we've conducted. It's what captures the ASOs' quality of service to their customers the best.

We also report the median transferSize as reported by the Navigation Timing API as a sanity check, to ensure that the RUM metrics comparison is fair between ASOs, and that the differences aren't caused by visitors using a particular ASO accessing much smaller or much bigger pages on average.

Privacy

In order to respect the privacy of our visitors, we only report ASOs for which we aggregate more than 500 unique pageviews to generate scores.

Service

How it works

The asoranking.py script run on the first of every month from an Analytics Team stat machine.

Puppet provisions and schedules this the Python script via Scap from a deployment server. It runs as a monthly cron job under the analytics-privatedata user (a requirement to have access to the Kerberos keytab).

The script reads queries Hive to read the Navigation Timing data that we originally collected from web browsers via EventLogging.

The reports are expected to be generated once a month. Our monitoring alert checks that the latest published dataset should be less than 32 days old. The alert also checks that the report is at least 1KB in size.

Output

When used as follows:

python asoranking.py --publish

This will output the generated ranking for the previous calendar month in the form of a TSV file, published to /srv/published-datasets/performance/autonomoussystems/.

As per Analytics/Web publication, these are publicly web-accessible at https://analytics.wikimedia.org/published/datasets/performance/autonomoussystems/.

This data is also periodically used for the web page at https://performance.wikimedia.org/asreport/ by the Performance Team (perfsite runbook).

See also

Further reading