View source for Performance/Fresnel

'''Fresnel''' is an automated tool for creating and comparing performance reports about web pages. It is used at Wikimedia Foundation to provide continuous feedback about the client-side performance of MediaWiki.
{{TOC|align=right}}

== Find result ==
Frensel runs on every patch submission to Gerrit for the master branch of mediawiki/core.git. The jenkins-bot user comments on the Gerrit change with a link to the Fresnel result.

[[File:Fresnel-201903-gerrit-pass.png|border|417x417px]][[File:Fresnel-201903-gerrit-fail.png|border|325x325px]]

=== Rerun ===
To run Fresnel on-demand, submit the phrase <code>''check perf''</code> as a Gerrit comment.

== Scenario comparisons ==

=== Recording ===
Fresnel is based on a concept of scenarios. Each scenario consists of a name, a URL, and a number of options that configure the browser.

For MediaWiki core, the scenarios are configured in [[git:mediawiki/core/+/master/.fresnel.yml|fresnel.yml]].

Each of these scenarios will be performed against a local MediaWiki install for the current patch. And, for the parent commit as well.

=== Comparison table ===
The "before" and "after" results of these scenarios are then compared and presented in the build output. Below are two examples.<gallery widths="400" heights="200">
File:Fresnel-201903-build-pass.png|Fresnel result showing a positive difference in transfer size, and page load time.
File:Fresnel-201903-build-fail.png|Fresnel result showing an increased cost for network transfers.
</gallery>
<pre style="overflow-x: auto; white-space: pre;">
### scenario View a page: navtiming

|----------------------------------------------------|---------------------|---------------------|--------|--|
| Metric                                             |              Before |               After |   Diff |  |
|----------------------------------------------------|---------------------|---------------------|--------|--|
| Time to first byte (responseStart)                 |     83 ms (± 34 ms) |     72 ms (± 26 ms) |   0 ms |  |
| Total page load time (loadEventEnd)                |    345 ms (± 40 ms) |    334 ms (± 35 ms) |   0 ms |  |
| Time from responseEnd to domComplete (processing)  |    254 ms (± 24 ms) |    248 ms (± 36 ms) |   0 ms |  |
| Time from loadEventStart to loadEventEnd (onLoad)  | <0.1 ms (± <0.1 ms) | <0.1 ms (± <0.1 ms) |   0 ms |  |
|----------------------------------------------------|---------------------|---------------------|--------|--|
</pre>
The comparison tables read as follows:

* Name of the scenario (e.g. "View a page", or "Load the editor").
* Name of the metric collector (e.g. "navtiming", "transfer", and "paint").

The "Before" and "After" columns show the aggregated result of the multiple repeat runs that Fresnel did for a given scenario. As of March 2019, Fresnel for MediaWiki repeats each scenario 5 times. The '''median''' is displayed with the variance between runs (as '''standard deviation''') in parenthesis. There is a small amount of variable by default for time-based metrics. This is mainly due to general variance in CPU tasks in modern computing given that the operating system and background processes needing a few interruptions as well. Also, the Jenkins worker nodes are hosted in Wikimedia Cloud VPS, where CPU and disk availability is shared with other virtual machines.

The difference tolerates 1 standard deviation by default. For example, a duration of  <code>200 ms (± 10 ms)</code> is considered as having a difference if the new value is below 190 ms or above 210 ms.

{{anchor|DevTools}}
== Artefacts ==
Fresnel records all of its data as JSON files are collected as artefacts by the Jenkins job and can be viewed for additional details.

In addition to these JSON files, Fresnel sends additional probes to the browser that collect other artefacts: [[#Performance Timeline|Timelines]], and [[#Screenshots|Screenshots]].

=== Find artefacts ===
<gallery widths="200" heights="120">
File:Fresnel-201903-build2status.png|On the result page, go to Status.
File:Fresnel-201903-status2artefacts.png|From Status, go to Build Artefacts.
File:Fresnel-201903-artefacts.png|Go to <code>log/fresnel_records</code>.
File:Fresnel-201903-record.png|File listing for one scenario-run.
</gallery>

=== Performance Timeline ===
If you have Chromium or Chrome Canary installed, then open the Chrome DevTools (e.g. on the Jenkins page itself) and drag the link to <code>trace--trace.json</code> onto it.
[[File:Fresnel-201903-timeline.png|none|thumb|608x608px|Open the trace log in the Performance tab of Chrome DevTools.]]

=== Screenshots ===
[[File:Fresnel-201903-record-screenshot.png|none|thumb|337x337px]]

== Run Fresnel locally ==
The [[#Artefacts|Artefacts]] being saved by the Jenkins job should generally forego the need to ''reproduce'' a build locally.  But, there's more to local use than merely reproducing the build.

Using Fresnel on your localhost allows you to iterate offline, and to compare any two recordings.

You can run Fresnel locally on any MediaWiki installation (consider a [[#Quick MediaWiki|quick install]]):

# Fresnel (from npm, Node.js 10+).
# An environment variable telling Fresnel about your MediaWiki install.

You can install Fresnel from npm.
 npm install fresnel@1.0.0
To run Fresnel, execute <code>fresnel record</code> from a MediaWiki core directory. Make sure you have the <code>MW_SERVER</code> the <code>MW_SCRIPT_PATH</code> environment variables set. You can set them for this terminal session only as shown below, but you may want to set them from your <code>~/.bashrc</code> file instead so they're always set. 

Once you have Fresnel installed, a typical workflow goes like this: 

* Record the baseline for your comparison. 
* Record the new state you want to compare to it. 
* Compare the two. 

Here's an example:
 $ MW_SERVER=<nowiki>http://default.web.mw.localhost:8080</nowiki> MW_SCRIPT_PATH=/mediawiki
 $ cd mediawiki/
 mediawiki$ git checkout master
 mediawiki$ fresnel record "my baseline"
 ..
 
 mediawiki$ # make some changes now, or checkout a different commit, etc.
 mediawiki$ fresnel record "new"
 ..
 
 mediawiki$ fresnel compare "my baseline" "new"
 
   scenario View a page: navtiming
   ..
== How it all works ==
When the Jenkins job for mediawiki-fresnel builds your patch, it essentially does the following:

# Checkout your patch from Gerrit. (with help of Zuul's Git Merger to support <code>Depends-On</code> etc.)
# Install MediaWiki as simply as possible. (This uses the PHP built-in server by running <code>php -S</code> in mediawiki/core; no Apache, database, or other setup; inspired by [[Mw:Continuous integration/Quibble|Quibble]]!)
# Play back each of the scenarios in Headless Chromium (using Puppeteer). There are warmups and repeat runs as well. This is configured in [[git:mediawiki/core/+/master/.fresnel.yml|fresnel.yml]].

=== Quick MediaWiki ===
Get MediaWiki up and running in '''under a minute'''! No Apache, MySQL, Docker, or Vagrant PHP required. Only PHP (install from your package manager, e.g. Homebrew).<syntaxhighlight lang="bash">
# Path from which to clone
export QMW_SRC="${MW_INSTALL_PATH:-https://gerrit.wikimedia.org/r/mediawiki/core}";
export QMW_DIR=/tmp/quickmw;
# For example:
# MW_INSTALL_PATH="$HOME/Development/mediawiki/core"
# Or, force getting it fresh from Gerrit:
# QMW_SRC="https://gerrit.wikimedia.org/r/mediawiki/core"

# Copy and execute as one command
test -n "$QMW_DIR" && rm -rf "$QMW_DIR" && mkdir -p "$QMW_DIR" && cd "$QMW_DIR" \
    && git clone --depth 1 -b master "$QMW_SRC" mediawiki && cd mediawiki/ && git clone --depth 1 https://gerrit.wikimedia.org/r/mediawiki/vendor && git clone --depth 1 https://gerrit.wikimedia.org/r/mediawiki/skins/Vector skins/Vector \
    && composer mw-install:sqlite \
    && php -S localhost:4000;
# …
# Listening on http://localhost:4000 !
</syntaxhighlight>New wiki should be up and running at http://localhost:4000.

What the above command does:

* Create a temporary clone of MediaWiki core. If you have a clone already, this should only take a few seconds. We also add vendor.
* Run <code>composer mw-install:sqlite</code>.
* Run <code>php -S localhost:4000</code>.

Log:
 Cloning into 'mediawiki'...
   Checking out files: 100%, done.
 Cloning into 'vendor'...
   Receiving objects: 100%, 2.32 MiB/s, done.
 The environment has been checked. You can install MediaWiki.
   Setting up database.. done
   Creating tables.. done
   Creating administrator user account.. done
   Creating main page with default content.. done
   MediaWiki has been successfully installed. 
 PHP 7.2 Development Server started
   Listening on http://localhost:4000
   Document root is /private/tmp/quickmw/mediawiki
   Press Ctrl-C to quit.
 ^C [runtime: '''17s'''] $

== Code steward ==
{{Steward|steward=[[mw:Wikimedia Performance Team|Performance Team]]|irc_channel=wikimedia-perf|phab=Fresnel}}

== See also ==

* Source code (Gerrit): [[git:performance/fresnel/|performance/fresnel.git]]
* [[Performance/Runbook/Fresnel]]

[[Category:Performance Team]]