You are browsing a read-only backup copy of Wikitech. The live site can be found at


From Wikitech-static
< Performance
Revision as of 20:24, 6 September 2018 by imported>Quiddity (fix header levels)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

WebPageTest is a web performance tool that uses real browsers to access web pages and collect timing metrics. The killer feature of WebPageTest is the metric called SpeedIndex – a measure of how fast the above-the-fold content is displayed. The Wikimedia Performance Team runs a private instance of WebPageTest at on AWS, and you can view the metrics we collect at


There are two standard ways of collecting timing metrics for web pages today: RUM (Real User Measurement) and synthetic testing. Collecting data using RUM means adding some JavaScript to pages. The script runs in users' browsers, collects some metrics (eg. Navigation Timing and User Timing), and beacons back the data to our servers. The upside of RUM metrics is that they are from real users, the downside is that it varies quite much and it's hard to isolate changes and correlate things. Furthermore, the metrics depend on latency and connection type. We are also missing good ways of measuring things that really matters like first paint (only in Chrome & IE11) and when content is finished loading within the current view port.

Synthetic testing on the other hand tries to minimize the different factors that can impact the metrics to let us pinpoint the correlation between code changes and metrics impact. Synthetic testing tries to run from the same location, same latency, same browser and measuring the same way. Using WebPageTest as our synthetic tool also has another advantage: SpeedIndex - the best way today to measure when the above the fold content is ready for the user. Synthetic testing's downside is that the metrics aren't from real users.

We use the NavigationTiming extension to add our own script to collect RUM metrics, we run a private instance of WebPageTest to collect synthetic testing and we run Browsertime/WebPageReplay to collect synthetic testing under isolated premisses.


The current setup looks like this:

WebPageTest setup


We have one instance at the moment called us-east. We don't use auto scaling since it's broken on Linux see

Install the CLI

We have use a small wrapper script that talks to the WebPageTest API using the It collects the metrics and reports the metrics as csv/json or to graphite/statsv.

You can clone the project or install the wrapper with

npm install wpt-reporter -g

Running a test using the CLI

If you have installed the project, you can simply run it like this:

wpt-reporter --webPageTestKey WPT_API_KEY --reporter json --webPageTestHost --location us-east:Chrome

It will send a request to our WebPageTest instance and start a test. Make sure to change the WPT_API_KEY value to your secret key.


If you want to see what you can configure and the default values, run:

wpt-reporter --help

All parameters that you send to the tool are passed to the WPT API, there are a lot of things you can configure and here is the full list.

Send to statsv

Sending statistics to statsv is disabled by default. Turn it on like this (by setting the reporter to statsv and setting a valid endpoint):

wpt-reporter --webPageTestKey WPT_API_KEY --endpoint YOUR_STATSV_ENDPOINT --reporter statsv --webPageTestHost --location us-east:Chrome

Store as CSV

If you want to verify that your changes are faster than your current version, you can use WebPageTest and test the changes and store the result as a CSV file. You can choose where CSV data will be stored with the file option. If the file doesn't exist, it will add one line with all the column names of the metrics. If the file exists, it will just append the new metrics on a new line.

wpt-reporter --webPageTestKey WPT_API_KEY --webPageTestHost --reporter csv --file myresult.csv --location us-east:Chrome

Mobile phone

If you want to test mobile pages, you can fake the user agent and set viewport and size using our own WebPageTest instance. If you want to test with a real mobile phone, you can use the Motorola G phones that are available on

Set view port and screen size

You can fake a mobile user agent (only when using) Chrome. By using emulateMobile you will use a Chrome mobile user agent, 640x960 screen, 2x scaling and fixed viewport.

wpt-reporter --webPageTestKey WPT_API_KEY --webPageTestHost --emulateMobile true --reporter json --location us-east:Chrome
Set your User Agent string

If you want to set your own user agent, use --userAgent (it will only work if you use Chrome).

wpt-reporter --webPageTestKey WPT_API_KEY  --webPageTestHost --location us-east:Chrome --userAgent "Mozilla/5.0(iPhone;U;CPUiPhoneOS4_0likeMacOSX;en-us)AppleWebKit/532.9(KHTML,likeGecko)Version/4.0.5Mobile/8A293Safari/6531.22.7" --reporter json
Use a real mobile phone

If you want to use a real mobile phone, you can do that by using the public instance of and use the Motorola 5. Make sure that you use the WebPageTest key for the public instance (and not the key for our private instance).

wpt-reporter  --webPageTestKey WPT_ORG_API_KEY --webPageTestHost --location "Dulles_MotoG:Motorola G - Chrome" --reporter json
Test with 2g connectivity

The current version of the WebPageTest API doesn't have a short handle for 2g but you can set the connectivity yourself and simulate 2g. Just make sure to also increase the timeout time, because running multiple tests on 2g takes time.

wpt-reporter --webPageTestKey WPT_API_KEY --webPageTestHost --bandwidthDown 35000 --bandwidthUp 32000 --latency 1300 --timeout 2400 --connectivity custom --reporter json --emulateMobile true --location us-east:Chrome

If you want to try out your own settings, you can checkout WebPageTest ini file for inspiration.

Test multiple URLs

You can test multiple URLs by choosing the --batch option. Running a batch will fetch all the URLs and the configuration from a file. You supply the path and name of the file and the file needs to have one URL/run on each line. It looks like this:

## Test the Facebook page 15 times
--webPageTestKey <%WMF_WPT_KEY> --webPageTestHost --runs 15 --median SpeedIndex --reporter json --location us-east:Chrome

## And then test Barack 31 and use SpeedIndex as median pick
--webPageTestKey <%WMF_WPT_KEY> --webPageTestHost --runs 31 --median SpeedIndex --reporter json --location us-east:Chrome

Look closely at the webPageTestKey value. The value <%WMF_WPT_KEY> will be replaced by the environment variable named WMF_WPT_KEY. Running in node, the value will be replaced with process.env.WMF_WPT_KEY. If the variable isn't found, there will be an error logged. You can create your own variables in the script files and feed the values with environment variables following the same pattern.

Changing and testing a batch file

When we run the tests in Jenkins we run batch scripts located here:

When you want to add a URL to test or change anything, you need to test it locally. You can easily do that by following the example in each batch.

Here's an example of running a batch locally.

WMF_WPT_KEY=OUR_SECRET_KEY STATSV_ENDPOINT= WPT_RUNS=1 WMF_WPT_LOCATION=us-east bin/index.js --batch scripts/batch/desktop.txt

When you add a new test, make sure to check out the screenshots on WebPageTest to see that it worked as expected (the user is logged in etc). There's also a bash script that will test all batch files, so if you make changes to the current batch scripts, please run it like this before you commit your changes:


When your changes has been approved and Jenkins pull them, make sure that the jobs in Jenkins work fine:

Run on Jenkins

We use Jenkins to continuously run the tests ( and Jenkins runs the batch files and to be able to do that, you need to do three things:

  • Git clone the project: and use the master branch refs/heads/master
  • Add the binding for the environment variables using Bindings and share them as Secret Text. You need to setup WMF_WPT_KEY, WPT_ORG_WPT_KEY, WPT_USER & WPT_USER_PASSWORD
  • Run the tests in an execute shell build step:
declare -i RESULT=0
# These tests runs on our own Linux WebPageTest instance
export WPT_RUNS="5"
export WPT_MOBILE_RUNS="5"
export WMF_WPT_LOCATION="us-east"
npm install --production
./bin/index.js --batch ./scripts/batch/mobile.txt
./bin/index.js --batch ./scripts/batch/desktop.txt
./bin/index.js --batch ./scripts/batch/login-mobile.txt
./bin/index.js --batch ./scripts/batch/login-desktop.txt
./bin/index.js --batch ./scripts/batch/second-view-mobile.txt
./bin/index.js --batch ./scripts/batch/second-view-desktop.txt
exit $RESULT

Find results

We automatically run wpt a couple of times a day to collect metrics and send them to statsv. If you want to look at a specific test, go to and choose Show tests from all users. You will then look at all the test runs for the last 30 days. You can change the time span by changing the View and choose Update list.

Test as a authenticated user (use scripting)

In the world of WebPageTest you can either test a specific URL or write a script that perform a couple of interactions; like access a page, log in the user and access another page. And you can choose when to start collecting metrics. You can read more about the setup:

Using scripts, it follows the same pattern as running batch scripts, meaning you can have variables in your script that will be replaced with the value of environment variables at run time. Name your variable <%YOUR_NAME_HERE> and it will be replaced with process.env.YOUR_NAME_HERE.

Using custom metrics

You can collect custom metrics using Javascript that will run in the browser when the page has been loaded. That's nothing we use today (meaning we don't support sending these metrics to statsv) but it's a pretty neat feature and we will have the data/metrics on the WebPageTest result pages.

How to test new features/changes

One idea with using WebPageTest is that we also can test new features, changes and have a good way of measuring metrics before and after a change. If you want to collect metrics for a change, it is important that you run the test many times so that we have a median value that can reflect the change. Many times means at least 31 times :) Always choose an odd number, that way a real run is picked as the median.

If you want to run a massive tests, you need to setup a new agent instance, else it will compete with our hourly runs.

Choose to report the result as CSV, then you will have one file with all the URLs you test in one condition, switch the condition and make another CSV and import the two CSV files into a program that handles CSV files. The CSV file will on each row contain the tested URL and the different metrics. This is a good approach:

  1. Create a batch file with the URLs you want to test. Make sure to parameterize things like connectivity, number of runs and so on. One line in your batch file can look like this:
    --webPageTestKey <%WMF_WPT_KEY> --webPageTestHost --median SpeedIndex --location <%WMF_WPT_LOCATION>:Chrome --label chrome --runs <%WPT_RUNS> --connectivity <%WPT_CONNECTIVITY> --reporter csv --file <%CSV_FILE_NAME%> --timeout <%WPT_TIMEOUT>
  2. Add the rest of the URLs that you want to test in the file by copy/pasting the row and changing the URL at the end.
  3. Setup your environment so you export the environment variables. First time your run it: make sure to only use one run and good connectivity like cable to run through the test fast and see that all URLs and configuration are ok. The timeout time is high by default but if you test with 2g for example, the test will take a long time to finish, so you need to increase the timeout value (default is 20 minutes and you define it in seconds, so default timeout is 1200).
  4. Verify the CSV file. Does it look the way you need it? Good, keep going.
  5. Change the number of runs to the number you want (and the connectivity) and run the script again. This can take time. Keep your terminal open, each and every test will be submitted when the last one is finished.
  6. Ok we have some numbers, now configure the change so that when you access the test URL we have the new feature/change. Make sure to change the name of the CSV file, so we keep two different ones.
  7. Run the test again.
  8. Now you have the numbers! Lets compare them. If you need help to understand them, talk to the performance team and we will help you analyze them.

Caution: Choose what to see on the result page

By default WebPageTest will pick the median run of pageLoadTime. That's not optimal because we want to focus on SpeedIndex or start render time. By adding parameters to the start result page, you can choose what run and metric that will picked up as the median run. Choose which metric to use and if you want the median or the fastest run:





WebPageTest and AWS

WebPageTest consists of two separate entities: a server and agent(s). On AWS there are ready made AMI:s (prepared images) for the two, so it is an easy to click and deploy.

WebPageTest can run headless or not. Headless in this context meaning no GUI available to start a test, you then need to use the API to submit tests.

Setup the server

It can be hard finding the right AMI, for a server in us-west we use AMI id ami-d7bde6e7

  1. Find the right AMI (ami-d7bde6e7) under Images/AMI and pick it (make sure to choose Public images).
  2. Choose Launch and use type t2.micro. Make sure to choose Next: Configure Instance Details
  3. Go the the Advanced Details section and add the configuration for the server. Make sure you change all the secret placeholders to the real values and choose Next: Choose storage
  4. Nothing you need to do here, choose Next: Tag Instance
  5. Add the tag Name with the value: WebPageTest Server and then Next: Configure Security Group
  6. Change the SSH access to only be our IP range.
  7. Add access for HTTP by choosing Add Rule - use the dropdown and choose HTTP and keep the rest of the values default.
  8. Choose Review and launch and then Launch. You will be asked to choose an existing key pair or use an existing. Create a new pair (name it webpagetest) and download it. You will need the keys to be able to SSH to the server so make sure to download them.
  9. Attach the new server the Elastic Path IP: NETWORK & SECURITY/Elastic IPs and choose the IP and Associate Address (we use ip
  10. Use the tag WebPageTest Server in the instance field and choose Associate.
  11. You should now be able to access and see the "headless" start page.

Login to the server

ssh -i webpagetest.pem


These are the configuration details that you use in the Advanced Details section.

; the key used when starting tests

; no GUI for submitting tests, but we can check the results

; Define maximums runs per URL

; Quality of images, lets define this to something good

; save png full-resolution screen shots

; automatically update the agent when a new version is available

; needed for autoscaling

; keep an instance up and running

; how long to keep tests locally before sending them to S3

; archiving to s3 (using the s3 protocol, not necessarily just s3)

Setup S3

WebPageTest can automatically store the test results on S3 and that is perfect for us so we can drop the server instance whenever we want.

To setup S3 (these are the instructions to do it the first time):

  1. Log into the AWS console and choose S3
  2. Choose Create a bucket
  3. Add a Bucket name and name it wpt-wikimedia (the bucket name needs to correspond to the property archive_s3_bucket when you configure the server).
  4. Add the Region. We use Oregon and that matches the configuration property archive_s3_server key.
  5. Choose Create and we have created the bucket.
  6. Next step is to setup the properties on the bucket, meaning giving access for HTTP traffic and the server to upload the test results.
  7. Choose your bucket (webpagetest) and choose Properties/Permissions.
  8. Choose Add more permissions and add Authenticated Users as Grantee and give it Upload/Delete permissions.
  9. Then we need to configure how long time we want to store the data.
  10. Choose Lifecycle and Add rule.
  11. Apply the rule for the whole bucket
  12. Choose Permanently Delete and 370 days.
  13. Choose Review and add a Rule Name: Permanently remove tests after 370 days
  14. Choose Create and Activate Rule

We are now finished setting up S3.

Setup EBS

WebPageTest is stateless and stores everything on file. To be able to find old tests, WebPageTest uses a log file. The log file is not backed up to S3, so to be able to find old tests if the server is dropped, we need to store the logs on a separate disk.

  1. Choose Elastic Block Store / Volumes
  2. Choose Create Volume
  3. Choose a Size in GiB (the lowest 30GB will do fine)
  4. Choose Availability Zone. Use the same as the server
  5. Leave everything else as the default and choose Create.
  6. Choose the radio button for the newly created volume and the Tag label.
  7. Add a new Tag with the key Name and the value WebPageTest logs and choose Save.
  8. Make sure the volume is selected with the radio button and choose Action/Attach Volume.
  9. Choose WebPageTest server as the instance and use the default Device and choose Attach.
  10. The volume is now attached to our server, the next step is to login to the server and make sure that the logs are stored on the device.
  11. Use the pem-file for the server and login: ssh -i NAME.pem ubuntu@SERVER_IP (change name of the pem file to your pem file and the SERVER_IP to the real IP and follow these instructions:
  12. Follow the instructions and mount the device to /data
  13. Now your volume is mounted, the next step is to change WebPageTest log dir to a symbolic link to a directory that exists on the volume. If you haven't done any tests, the directory should be empty except for a .htaccess file.
  14. Make your new directory on the mounted device: sudo mkdir /data/logs
  15. Move the access file: sudo mv /var/www/webpagetest/www/logs/.htaccess /data/logs
  16. Remove the old one: sudo rm -fR /var/www/webpagetest/www/logs/
  17. Make the symbolic: sudo ln -s /data/logs /var/www/webpagetest/www/logs
  18. Make sure we have the right owner for the directory: sudo chown -h www-data:www-data /data/logs

Connectivity profiles

Depending on the AMI image, it could be that we are missing connectivity profiles: 3GFast, 3GSlow and 2G. If they are missing, you should add them in /var/www/webpagetest/www/settings/connectivity.ini

label="Mobile 3G - Fast (1.6 Mbps/768 Kbps 150ms RTT)"

label="Mobile 3G - Slow (780 Kbps/330 Kbps 200ms RTT)"

label="Mobile 2G (280 Kbps/256 Kbps 800ms RTT)"


The username and password for the master AWS account is recorded in iron:/srv/passwords/aws-webpagetest. Please avoid using this account directly. Ask an existing maintainer to create an IAM user for you instead. The IAM users sign-in link for this account is .


If something isn't working for you on the WebPageTest server instance you can find the logs here (yep they are on different locations)


Restart the server

If for some reason you want to restart the server (normally if you manually changed settings in /var/www/webpagetest/www/settings/settings.ini) restart nginx:

sudo service nginx restart

Archive old tests

Old test data will automatically be sent to S3. But we also need to remove old tests, easiest way to do it is to run the archive page. We do it in the crontab. Edit the crontab (as the ubuntu user): crontab -e

0 * * * * curl -sS >> /tmp/cron.txt

We then run archiving every our.

Cloud watch

We use Amazon cloud watch to keep track of the disk space of the WebPageTest server. You need to install a couple if libraries to get it up and running, follow the instructions. Then add one line to your crontab to start sending the metrics to Amazon.

*/5 * * * * ~/aws-scripts-mon/ --disk-space-used --disk-space-avail --disk-path=/ --from-cron

Then setup an alarm for the disk space. The current alarm warns (sends an email to the web perf list) when we only have 2 gb disk free.

Install the wrapper on the server

You only need to do this if you don't want to run the wrapper on Jenkins. First we install node & git to be able to get and run our wrapper:

sudo apt-get update && sudo apt-get install -y git nodejs npm && sudo ln -s /usr/bin/nodejs /usr/local/bin/node

Install the latest version of the wrapper

npm install wpt-reporter -g

Add jobs to run automatically

To schedule jobs we use Jenkins.

Setup the agents

We have an agent up and running on us-east-1 on EC2 and it is called us-east. That's the one that we will use for now and if you need to setup your own instance, you need to do like this:

Illustration of Step 3.
  1. In the AWS console, choose to Launch instance and choose Community AMIs. There you can find the prepared AMI. For us-east-1 I used ami-a88c20d5. You can find all prepared AMIs here (you need a specific per location).
  2. Choose the instance size c4.large.
  3. Then you need to make sure that your instance talk to the WebPageTest server, you do that by adding the configuration in the Advanced Details text field. The SECRET key is available on your WebPageTest server. wpt_loc=us-east wpt_url= wpt_key=SECRET

Then you can start your agent.

The next step is to configure the WebPageTest server so that it knows about the agent. You do that in /var/www/webpagetest/www/settings/locations.ini. That file is parsed with the ec2_locations.ini file and the result is the configured agents.


label="Linux US east 1"

browser=Chrome,Chrome Beta,Chrome Canary,Firefox,Firefox Nightly,Opera,Opera Beta,Opera Developer
label="Linux US east"

You can read more about how to configure the locations.ini file at

When you started your agent, and changed the locations.ini file, restart nginx on the WebPageTest server:

sudo service nginx restart

And then verify that you can see your instance at

Connect to an agent

You can ssh to the agent with the WebPageTestAgent.pem file. You can find the IP of the agent on AWS.

Timeout and agents not responding

We have seen that a couple of times one of the agents just stop working. You can see that by that all tests timeout and if you go to and check the latest finished results, the report will say that the agent couldn't be contacted by the server. To fix that, you need to login to the AWS console and go to EC2 management and make sure you are on "US East" region, choose the agent (it is named WebPagetest Agent), and choose Instance state -> Restart.

Bulk test

If you want to test changes before and after it's super important to test it many times to get correct values, use WPTBulkTest for that. Make sure to setup a new agent for your bulk test!


At the moment our test instance is busy running our continuously performance tests that we graph on We run one test agent to minimize the costs. If you want to use to run your own one shot tests, I ( can help you with that. You will need the key for the instance and choose which location you wanna use and then I can help you verify that the location is setup with the correct instance type.

Alert setup

We run automatic tests every hour (you can find the tests here We test mainly test the English Wikipedia: 3 desktop URLs using Chrome, the same URLs using Firefox and the mobile version on emulated mobile. You can see how we graph the metrics (and alert on regression):

Our mainly focus is testing on empty browser cache but we also run test with multiple page views (first hit one and then another) and as authenticated users. The metrics are too unstable at the moment to add alerts but we hope we can do that in the future.