You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Performance/AS Report: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Krinkle
(Mostly copied from the README of https://gerrit.wikimedia.org/g/performance/asoranking)
 
imported>Krinkle
No edit summary
 
Line 7: Line 7:
Our CPU microbenchmark ([https://github.com/wikimedia/mediawiki-extensions-NavigationTiming/blob/bfa8e6dc29ef32d69bc32dc66ab1855f7f9fd3f7/modules/ext.navigationTiming.js#L397-L412 source code]) is run for a short time in a separate [https://html.spec.whatwg.org/multipage/workers.html#worker Worker] thread, to avoid disrupting their experience. The benchmark assesses the overall performance of the device at the time of the measurement. This is important, as background memory usage and battery levels can greatly influence the performance of mobile devicess at any given time. In the wild, we have observed wide ranges of CPU scores for the exact same device model.
Our CPU microbenchmark ([https://github.com/wikimedia/mediawiki-extensions-NavigationTiming/blob/bfa8e6dc29ef32d69bc32dc66ab1855f7f9fd3f7/modules/ext.navigationTiming.js#L397-L412 source code]) is run for a short time in a separate [https://html.spec.whatwg.org/multipage/workers.html#worker Worker] thread, to avoid disrupting their experience. The benchmark assesses the overall performance of the device at the time of the measurement. This is important, as background memory usage and battery levels can greatly influence the performance of mobile devicess at any given time. In the wild, we have observed wide ranges of CPU scores for the exact same device model.


Using this information, we only keep the client metrics for a range of CPU scores. We then compile medians for that device range for each [[w:Autonomous system (Internet)|autonomous system]], based on the visitors' IP addresses ([https://github.com/wikimedia/performance-asoranking source code]).
Using this information, we only keep the client metrics for a range of CPU scores. We then compile medians for that device range for each [[w:Autonomous system (Internet)|autonomous system]], based on the visitors' IP addresses.


=== Mobile vs desktop ===
=== Mobile vs desktop ===
Line 30: Line 30:
In order to respect the privacy of our visitors, we only report ASOs for which we aggregate more than 500 unique pageviews to generate scores.
In order to respect the privacy of our visitors, we only report ASOs for which we aggregate more than 500 unique pageviews to generate scores.


== Usage ==
== Service ==
This script is meant to be run on a stat machine on cron on the 1st of every month, using the following syntax:


<code>python asoranking.py --publish</code>
* Source code: [[git:performance/asoranking|performance/asoranking.git]].
* Code review: [https://gerrit.wikimedia.org/r/q/project:performance%252Fasoranking Gerrit: performance/asoranking]


Which will generate the ranking for the previous calendar month in the form of a tsv file published to /<code>srv/published-datasets/performance/autonomoussystems/</code>
* Puppet class: [[git:operations/puppet/+/50f69b06872950db5646a428e5d77d1b85a6cf52/modules/statistics/manifests/performance.pp|statistics::performance]].
* Current host: [[stat1007]] (controlled by <code>profile::statistics::explorer::misc_jobs::hosts_with_jobs</code> in Hiera).


This is in turn picked up periodically by the Performance Team when regenerating [[Performance/Runbook/Webperf-processor services#site|performance.wikimedia.org]].
=== How it works ===
The <code>asoranking.py</code> script run on the first of every month from an Analytics Team [[stat machines|stat machine]].
 
Puppet provisions and schedules this the Python script via [[Scap]] from a [[deployment server]]. It runs as a monthly cron job under the analytics-privatedata user (a requirement to have access to the [[Analytics/Systems/Kerberos|Kerberos]] keytab).
 
The script reads queries [[Hive]] to read the Navigation Timing data that we originally collected from web browsers via EventLogging.
 
The reports are expected to be generated once a month. Our monitoring alert checks that the latest published dataset should be less than 32 days old. The alert also checks that the report is at least 1KB in size.
 
=== Output ===
When used as follows:
python asoranking.py --publish
This will output the generated ranking for the previous calendar month in the form of a TSV file, published to <code>/srv/published-datasets/performance/autonomoussystems/</code>.
 
As per [[Analytics/Web publication]], these are publicly web-accessible at https://analytics.wikimedia.org/published/datasets/performance/autonomoussystems/.
 
This data is also periodically used for the web page at https://performance.wikimedia.org/asreport/ by the Performance Team ([[Performance/Runbook/Webperf-processor services#site|perfsite runbook]]).  
 
== See also ==
 
* [[Analytics/Web publication]]
* [[Performance/Runbook/Webperf-processor services#site]]
* [[Performance/Metrics#navtiming]]


== Further reading ==
== Further reading ==
Line 44: Line 67:


* [https://techblog.wikimedia.org/2019/03/27/autonomous-systems-performance-report/ Introducing: Autonomous Systems report] (2019), by Gilles Dubuc, techblog.wikimedia.org.
* [https://techblog.wikimedia.org/2019/03/27/autonomous-systems-performance-report/ Introducing: Autonomous Systems report] (2019), by Gilles Dubuc, techblog.wikimedia.org.
[[Category:Performance Team]]
[[Category:Services]]

Latest revision as of 03:53, 30 June 2022

The AS Report (Autonomous Systems performance report) is a monthly-generated dataset anonymised from sampled visits to Wikipedia and other Wikimedia Foundation sites. The ranking is generated based on normalized RUM navtiming metrics measured in the web browser from sampled page views.

Methodology

Performance metrics are collected using the Navigation Timing API.

CPU microbenchmark

Our CPU microbenchmark (source code) is run for a short time in a separate Worker thread, to avoid disrupting their experience. The benchmark assesses the overall performance of the device at the time of the measurement. This is important, as background memory usage and battery levels can greatly influence the performance of mobile devicess at any given time. In the wild, we have observed wide ranges of CPU scores for the exact same device model.

Using this information, we only keep the client metrics for a range of CPU scores. We then compile medians for that device range for each autonomous system, based on the visitors' IP addresses.

Mobile vs desktop

We separate mobile and desktop experiences, as they have significantly different median page weights. As such, scores aren't comparable between mobile and desktop for a given country. Just as they are not comparable between countries either, due to different CPU benchmark score slices being selected for each. Reports are per country, as Internet services tend to be sold to consumers on a national basis, and networks in the same country face the same infrastructural challenges.

Mobile networks are those with at least one sampled visit having a "cellular" connection type, as reported by the Network Information API. When calculating the scores for mobile, we only consider data from sampled pageviews to the mobile site. Desktop networks are those with at least one sampled visit having a "wifi" connection type. When calculating scores for desktop, we only consider data from desktop pageviews.

We don't want an AS corresponding to an ISP selling femtocell devices widely to be unfairly advantaged or disadvantaged due to differences between the mobile and desktop experience, resulting in lighter or heavier pages on average.

Only AS organizations for which we found sufficient data are retained. Finally, only countries with sufficient data from sampled page views are reported.

RUM metrics

We report the medians for 2 core RUM metrics measures by the visitors' browsers: Time to first byte and Page Load Time, collected using the Navigation Timing API.

Time to first byte (TTFB) is how long it takes between the client requesting the page and it receiving the first byte of data from us. Page load time (PLT) is how long it takes to load the whole page, including all images and critical styles/scripts.

TTFB is the metric closest to latency, which is something ASOs might improve by peering with us or tweaking their routes to us. PLT is the metric that correlates the most to the visitors' perception of performance, as shown by research we've conducted. It's what captures the ASOs' quality of service to their customers the best.

We also report the median transferSize as reported by the Navigation Timing API as a sanity check, to ensure that the RUM metrics comparison is fair between ASOs, and that the differences aren't caused by visitors using a particular ASO accessing much smaller or much bigger pages on average.

Privacy

In order to respect the privacy of our visitors, we only report ASOs for which we aggregate more than 500 unique pageviews to generate scores.

Service

How it works

The asoranking.py script run on the first of every month from an Analytics Team stat machine.

Puppet provisions and schedules this the Python script via Scap from a deployment server. It runs as a monthly cron job under the analytics-privatedata user (a requirement to have access to the Kerberos keytab).

The script reads queries Hive to read the Navigation Timing data that we originally collected from web browsers via EventLogging.

The reports are expected to be generated once a month. Our monitoring alert checks that the latest published dataset should be less than 32 days old. The alert also checks that the report is at least 1KB in size.

Output

When used as follows:

python asoranking.py --publish

This will output the generated ranking for the previous calendar month in the form of a TSV file, published to /srv/published-datasets/performance/autonomoussystems/.

As per Analytics/Web publication, these are publicly web-accessible at https://analytics.wikimedia.org/published/datasets/performance/autonomoussystems/.

This data is also periodically used for the web page at https://performance.wikimedia.org/asreport/ by the Performance Team (perfsite runbook).

See also

Further reading