You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Performance/Synthetic Measurement Experiment 2019

From Wikitech-static
Jump to navigation Jump to search

Background

The current testing we do is mainly focused on testing Wikipedia with a cold browser cache: for each URL we test, we start the browser with a new session. We also have a couple of tests where we log in the user but we have never focused on what's the difference in performance if a user is logged in vs logged out vs a user that has some of the assets for the pages in the cache/local storage.

Goal

The main goal for this experiment it to share some light on the difference user experience if you are logged in vs logged out vs assets in cache. It is good to have one reference point so that we can compare and do the same in the future.

Test setup

We run the test on AWS. We use c5.xlarge (that the same size we use in the other synthetic testing). One server is located in India (Mumbai) and one in Sweden. We run the test in Chrome (v72) and for the mobile URLs we run Chrome in emulated mobil mode for a iPhone 6. The test was done in https://phabricator.wikimedia.org/T215489

Test scenarios

We run the same test scenarios from both places but with different URLs:

  • We access five URLs after each other, where we close the browser between each URL and start a new browser session with a cleared cached between each URL.
  • We access the same five URLs after each other, waiting 20 s between we hit each page with the browser. We don't close the browser between each URL so the browser cache can be filled. This way we want to simulate a user reading a couple of pages at Wikipedia.
  • We login the user and then access all five URLs after each other (keeping the browser open). We wait 20 s between each URL.

Tested URLs

We test English URLs from India since we get a lot of traffic to the English version. From Sweden we test the Swedish Wikipedia. We used a couple of pages: some are more light weight, some are more heavy weight.

India

Desktop
  • https://en.wikipedia.org/wiki/Main_Page
  • https://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2018
  • https://en.wikipedia.org/wiki/Goods_and_Services_Tax_(India)
  • https://en.wikipedia.org/wiki/List_of_Chief_Ministers_of_Uttar_Pradesh
  • https://en.wikipedia.org/wiki/India
Mobile
  • https://en.m.wikipedia.org/wiki/Main_Page
  • https://en.m.wikipedia.org/wiki/List_of_Bollywood_films_of_2018
  • https://en.m.wikipedia.org/wiki/Goods_and_Services_Tax_(India)
  • https://en.m.wikipedia.org/wiki/List_of_Chief_Ministers_of_Uttar_Pradesh
  • https://en.m.wikipedia.org/wiki/India

Sweden

Desktop
  • https://sv.wikipedia.org/wiki/Portal:Huvudsida
  • https://sv.wikipedia.org/wiki/Astrid_Lindgren
  • https://sv.wikipedia.org/wiki/Sm%C3%A5land
  • https://sv.wikipedia.org/wiki/Stockholm
  • https://sv.wikipedia.org/wiki/Huvudstad
Mobile
  • https://sv.m.wikipedia.org/wiki/Portal:Huvudsida
  • https://sv.m.wikipedia.org/wiki/Astrid_Lindgren
  • https://sv.m.wikipedia.org/wiki/Sm%C3%A5land
  • https://sv.m.wikipedia.org/wiki/Stockholm
  • https://sv.m.wikipedia.org/wiki/Huvudstad

Metrics

We collect a lot of different metrics:

  • Visual metrics - metrics that are collected when analyzing a video of the screen. Here we can see when the first/last visual change happens within the viewport, when the largest image is painted.
  • CPU time spent metrics - we collect where Chrome spend time: parsing Javascript, rendering and painting the screen
  • Transfer size - how much data each page send of the internet wire.
  • Amount of requests - how many requests that is used for each different test

Result


Let us go through the results and see if we can find any patterns. Remember that we only tested at two different connectivity types and a limited set of pages.

Slow connection (3g) mobile in India

We test a couple of different URLs, some URLs need to do more requests and some fewer. We start by taking a look at https://en.m.wikipedia.org/wiki/Goods_and_Services_Tax_(India) that got most request cached when the user has accessed another Wikipedia page before. The first visual change looks like this:

First visual change

Here we can see that users that are not logged in but have visited pages before are the fastest. We can also see that logged in users have a faster first visual change than a user that with a cold cache.

How much differs the data that is downloaded? The page is pretty lightweight and difference looks like this.

Transfer size and requests Goods and Services Tax mobile.png

And the difference in actual JavaScript and CSS:

Goods and services JavaScript and CSS

Another way of looking at the metrics is visual progress of the viewport. When 100% is painted the user see everything in the viewport. This is what the progress looks like if we compare a total cold cache (green) with some items cached (red).

Visual progress using 3g from Mumbai en.m.wikipedia.org/wiki/Goods_and_Services_Tax_(India)

And then comparing logged users (green) with a total cold cache (red).

Visual progress using 3g from Mumbai en.m.wikipedia.org/wiki/Goods_and_Services_Tax_(India)

And then finally logged users (green) and users with a warm cache (items in the cache).

Visual progress using 3g from Mumbai


If we look at another page https://en.m.wikipedia.org/wiki/List_of_Bollywood_films_of_2018 that makes more requests the Visual progress looks like this (mobile is on the right):

List of Bollywood films 2018

For mobile it is the same pattern.

Transfer size and requests Bollywood mobile.png
JavaScript and CSS transfer size Bollywood mobile.png

And the progress (green is cold cache and red logged in):

Visual Progress using 3g from Mumbai en.m.wikipedia.org/wiki/List_of_Bollywood_films_of_2018

Cable connection desktop in India

The desktop site has an other patterns for first visual change: Warm cache is always fastest and then sometimes logged in is faster and sometimes no cache.

First Visual Change desktop Goods And Services
First Visual Change desktop Bollywood
First Visual Change desktop India

And the visual progress looks like this:

Goods and Services
Goods and Services
Goods And Services

Slow connection (3g) mobile in Sweden

On mobile from Sweden we see the exact same behavior on all pages on mobile as from India. Warm cache is always fastest, then a logged in user (with a warm cache) and then cold cache.

Stockholm


Cable connection desktop in Sweden

On desktop we also see the same thing as with the tests from India. Warm cache always fastest. Logged in/cold cache depends on the page.

Småland Desktop
Stockholm Desktop


Summary

Here are the patterns we can see with the testing we done so far:

  • A user that is not logged in and have visited other Wikipedia pages before, always have the fastest experience. This is true on both mobile and desktop.
  • On mobile a logged in user has a better experience than a user that has a cold cache.
  • On desktop a logged in user and a user with cold cache seems to have roughly the same experience.

Remember that we only tested at two different connectivity types and a limited set of pages. We should try to see what kind of patterns we can find with the users we collect from real users. Also note that we emulated the mobile by using a desktop browser in emulated mode, that is not the same as running the same tests on a real mobile phone.

Raw results

Sweden First Visual Change

First Visual Change Stockholm
First Visual Change Småland


First Visual Change Huvudstad


First Visual Change Astrid Lindgren


Page weight and requests desktop Sweden

Stockholm Desktop
Småland Desktop
Huvudstad Desktop
Huvudstad Desktop


Page weight and requests mobile Sweden

Stockholm Mobile
Småland Mobile
Huvudstad Mobile
Astrid Lindgren Mobile


Measuring the login step Sweden

Login Sweden

India First Visual Change

Goods and services tax
India


List of Bollywood films 2018
List of Chief Minister Uttar


Page weight and requests desktop India

Goods and Services transfer size and requests


Goods and services JavaScript and CSS
Transfer and requests India
India JavaScript and CSS transfer size.
Transfer size and requests Bollywood.png
JavaScript and CSS transfer size Bollywood.png


Transfer size and requests Chief Ministers of Uttar.png
JavaScript and CSS transfer size Chief Ministers of Uttar.png


Page weight and requests mobile India

Transfer size and requests Goods and Services Tax mobile.png
JavaScript and CSS transfer size Goods And Services Tax.png
Transfer and requests India mobile.png
JavaScript and CSS transfer size India mobile.png


Transfer size and requests Bollywood mobile.png
JavaScript and CSS transfer size Bollywood mobile.png
Transfer size and requests Chief Ministers of Uttar mobile .png
JavaScript and CSS transfer size Chief Ministers of Uttar mobile.png


Measuring the login step India

Login

Scripts

Hitting URLs with a cache:

module.exports = async function (context, commands) {
    await commands.measure.start('https://en.wikipedia.org/wiki/Main_Page');
    await commands.wait.byTime(20000);
    await commands.js.run('document.body.innerHTML = ""; document.body.style.backgroundColor = "white";');
    await commands.measure.start('https://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2018 ');
    await commands.wait.byTime(20000);
    await commands.js.run('document.body.innerHTML = "";document.body.style.backgroundColor = "white";');
    await commands.measure.start('https://en.wikipedia.org/wiki/Goods_and_Services_Tax_(India)');
    await commands.wait.byTime(20000);
    await commands.js.run('document.body.innerHTML = ""; document.body.style.backgroundColor = "white";');
    await commands.measure.start('https://en.wikipedia.org/wiki/List_of_Chief_Ministers_of_Uttar_Pradesh ');
    await commands.wait.byTime(20000);
    await commands.js.run('document.body.innerHTML = ""; document.body.style.backgroundColor = "white";');
    await commands.measure.start('https://en.wikipedia.org/wiki/India');
};

Hitting pages as a logged in user:

module.exports = async function (context, commands) {
    await commands.navigate(
        'https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page '
    );

    try {
        // Add text into an input field, finding the field by id
        await commands.addText.byId('USERNAME', 'wpName1');
        await commands.addText.byId('PASSWORD', 'wpPassword1');

        // Start the measurement and give it the alias login
        // The alias will be used when the metrics is sent to
        // Graphite/InfluxDB
        await commands.js.run('document.body.style.display = "none";');
        await commands.wait.byTime(1000);
        await commands.measure.start('login');

        // Find the submit button and click it and wait for the
        // page complete check to finish on the next loaded URL
        await commands.click.byIdAndWait('wpLoginAttempt');
        // Stop and collect the metrics
        await commands.measure.stop();
        await commands.wait.byTime(20000);
        await commands.js.run('document.body.style.display = "none";');
        await commands.measure.start('https://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2018');
        await commands.wait.byTime(20000);
        await commands.js.run('document.body.style.display = "none";');
        await commands.measure.start('https://en.wikipedia.org/wiki/Goods_and_Services_Tax_(India)');
        await commands.wait.byTime(20000);
        await commands.js.run('document.body.style.display = "none";');
        await commands.measure.start('https://en.wikipedia.org/wiki/List_of_Chief_Ministers_of_Uttar_Pradesh');
        await commands.wait.byTime(20000);
        await commands.js.run('document.body.style.display = "none";');
        return commands.measure.start('https://en.wikipedia.org/wiki/India');
    } catch (e) {
    }
};