You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Performance/Synthetic testing/Mobile phones: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Phedenskog
(Added photo of device lab.)
imported>Phedenskog
(Added temperature issues.)
Line 79: Line 79:
</syntaxhighlight>And the ''secrets.json'' file contains configuration to sitespeed.io to be able to send metrics to our Graphite instance and send data to S3. All tests then extends that configuration file and we can have those configurations file open to the public in our Git repo.
</syntaxhighlight>And the ''secrets.json'' file contains configuration to sitespeed.io to be able to send metrics to our Graphite instance and send data to S3. All tests then extends that configuration file and we can have those configurations file open to the public in our Git repo.


We then have cronjob calling the API using CURL with settings to use our zip file and sending one extra parameter that will choose what test to run.
We then have cronjob calling the API using CURL with settings to use our zip file and sending one extra parameter that will choose what test to run. At the moment the job is fired from ''wpt-runner.webperf.eqiad1.wikimedia.cloud'' but lets change that in the future,


== Performance tests ==
== Performance tests ==
Line 94: Line 94:


== Troubleshooting ==
== Troubleshooting ==
=== The device is offline ===
All phones trouble shooting is handled by BitBar. If one phone is ''"offline"'' or ''"online dirty"'' we contact BitBar on using the BitBar/Wikimedia Slack channel.
If the device is offline you can see that in the log, look for ''The phone state is offline.'' You can also verify that by running <code>adb devices</code>. Talk with the Wikimedia Performance team and we will contact Kobiton.
 
=== Missing data for a phone ===
If you see in the graphs in Grafana that data is missing you need to check the logs on the server. Each phone has their own log. Say that the phone ''ZY322HX8RC-Moto-G5'' has the problem, then go to the directory ''performance-mobile-synthetic-monitoring-tests/logs'' and check the log file named '''ZY322HX8RC-Moto-G5.log.''' Look for error messages in the log file.


== Dashboards ==
== Dashboards ==
We have three dashboards:
The data is reported under the '''android''' key in Graphite. Make sure ''type'' is '''android''' in the [https://grafana.wikimedia.org/d/IvAfnmLMk/page-drilldown?var-base=sitespeed_io&var-path=android&var-testtype=webpagereplay&var-group=en_m_wikipedia_org&var-page=_wiki_Barack_Obama&var-browser=chrome&var-connectivity=custom&var-function=median&var-s3path=https:%2F%2Fsynthetic-tests-result-wikimedia.s3.amazonaws.com dashboard].


== Outstanding issues ==
== Outstanding issues ==
There are a couple of issues that we need to fix in the current setup
At the moment we are looking into high battery temperature (high means over 35 degrees Celsius) on some of the Moto G5 phones.

Revision as of 13:35, 3 October 2022

Summary

In the path to get more realistic performance metric run tests on real mobile devices. That makes it easier to find performance regressions. We use BitBar as the provider of our devices. All tasks are reported under the Performance Device Lab tag in Phabricator.

Performance testing on mobile phones

Running tests on mobile phones we want a stable environment that do not change, so we can see performance regressions. To make that happen we use:

  • A stable network: We throttle the connection to look like a 3g or 4g connection.By limiting the upload/download speed and adding delay, we make the phone get requests in a more realistic scenario. By making sure the network is the same all the time, the network will not affect our metrics.
  • Low phone temperature: We measure the battery temperature as a proxy for CPU temperature. Android phones change behavior when the CPU gets warm and we want to avoid that. Some of our phones are rooted to try to make sure the phone has the same performance characteristics. We use the same settings as the Mozilla performance team to setup the rooted phones. We measure the temperature before and after we start a test. If the temperature is too high we wait X minutes and try again.

Setup

We use five mobile phones, a server and two wifi:s setup with throttled connection to simulate 3g and 4g traffic. The wifi connections is provided by two Raspberry Pis 4 running humble.

Setup showing the mobile performance device lab.

The workflow: The jobs are started on the server that runs sitespeed.io that drives the phones using WebDriver. The configuration and URLs to tests exists in a public Git repo. The tests runs on the phones, access Wikipedia and we record a video of the screen and analyze the result to get visual metrics. The metrics is sent to our Graphite instance and the test results (screenshots, HTML result pages) are stored on S3. We also run one tests using WebPageReplay where we record and replay Wikipedia locally on the server to try to get as stable metrics as possible between runs.

File:Performance-device-lab-at-bitbar.jpg
Performance device lab at BitBar.

Setup the phones

BitBar is handling the setup of phones. If we need to change anything we need to contact them.

We have five phones running at the Performance Device Lab with the following setup.

Id Type Internet connection Extras OS version Usage
ZY322DJBW9 Motorola Moto G5 #1 Simulated 3g Root 8.1.0 User journey: login
ZY322GXQN8 Motorola Moto G5 #2 Simulated 4g Root 8.1.0 First views and second views in user journey
ZY322H9XKL Motorola Moto G5 #3 Simulated 4g Root 8.1.0
R58NC31FK3Y Samsung A51 WebPageReplay Root 10 Running against WebPageReplay
Apple iPhone 11 Simulated 3g 15.2.1

Using rooted phones makes it possible to stabilise CPU and GPU performance by configuring governors.

BitBar setup

At BitBar our test use a generic setup with a start bash file (run-tests.sh) and a secrets.json file that are uploaded in the BitBar GUI in a zip file. The bash file is called when a test is started and looks like this:

# We unpack tests.zip that contains the secrets.json configuration file
unzip tests.zip

# Clone the git repo where we have all the code
git clone https://github.com/wikimedia/performance-mobile-synthetic-monitoring-tests.git
cd performance-mobile-synthetic-monitoring-tests

# There's a hack on BitBar where you can pass on a parameter
echo "The first is $1"
./start.sh $1

And the secrets.json file contains configuration to sitespeed.io to be able to send metrics to our Graphite instance and send data to S3. All tests then extends that configuration file and we can have those configurations file open to the public in our Git repo.

We then have cronjob calling the API using CURL with settings to use our zip file and sending one extra parameter that will choose what test to run. At the moment the job is fired from wpt-runner.webperf.eqiad1.wikimedia.cloud but lets change that in the future,

Performance tests

All configuration and setups for our tests lives in Gerrit in performance/mobile-synthetic-monitoring-tests. To add or change tests you clone the repo and send in your change for review.

Add a test

All configuration files exists in our synthetic monitoring tests repo. Clone the repo and go into the tests folder:

git clone ssh://USERNAME@gerrit.wikimedia.org:29418/performance/mobile-synthetic-monitoring-tests.git

cd mobile-synthetic-monitoring-tests/tests

Change configuration

Troubleshooting

All phones trouble shooting is handled by BitBar. If one phone is "offline" or "online dirty" we contact BitBar on using the BitBar/Wikimedia Slack channel.

Dashboards

The data is reported under the android key in Graphite. Make sure type is android in the dashboard.

Outstanding issues

At the moment we are looking into high battery temperature (high means over 35 degrees Celsius) on some of the Moto G5 phones.