You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Performance/Synthetic testing/Mobile phones: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Phedenskog
(Preparing for adding the new BitBar setup)
imported>Phedenskog
(Added photo of device lab.)
Line 9: Line 9:


== Setup ==
== Setup ==
We use five mobile phones, a server and two wifi:s setup with throttled connection to simulate 3g and 4g traffic.
We use five mobile phones, a server and two wifi:s setup with throttled connection to simulate 3g and 4g traffic. The wifi connections is provided by two Raspberry Pis 4 running [https://github.com/sitespeedio/humble humble].
[[File:Performance Device Lab setup.png|center|frame|Setup showing the mobile performance device lab.]]The workflow: The jobs are started on the server that runs sitespeed.io that drives the phones using WebDriver. The configuration and URLs to tests exists in a public Git repo. The tests runs on the phones, access Wikipedia and we record a video of the screen and analyze the result to get visual metrics. The metrics is sent to our Graphite instance and the test results (screenshots, HTML result pages) are stored on S3. We also run one tests using WebPageReplay where we record and replay Wikipedia locally on the server to try to get as stable metrics as possible between runs.  
[[File:Performance Device Lab setup.png|center|frame|Setup showing the mobile performance device lab.]]The workflow: The jobs are started on the server that runs sitespeed.io that drives the phones using WebDriver. The configuration and URLs to tests exists in a public Git repo. The tests runs on the phones, access Wikipedia and we record a video of the screen and analyze the result to get visual metrics. The metrics is sent to our Graphite instance and the test results (screenshots, HTML result pages) are stored on S3. We also run one tests using WebPageReplay where we record and replay Wikipedia locally on the server to try to get as stable metrics as possible between runs.
 
[[File:Performance-device-lab-at-bitbar.jpg|center|thumb|600x600px|Performance device lab at BitBar.]]


=== Setup the phones ===
=== Setup the phones ===
BitBar is handling the setup of phones.  
BitBar is handling the setup of phones. If we need to change anything we need to contact them.  


We have five phones running at the Performance Device Lab with the following setup.
We have five phones running at the Performance Device Lab with the following setup.
Line 25: Line 27:
!Usage
!Usage
|-
|-
|ZY2242HJDX
|ZY322DJBW9
|Moto G4
|Motorola Moto G5 #1
|Simulated 3g
|Root
|8.1.0
|User journey: login
|-
|ZY322GXQN8
|Motorola Moto G5 #2
|Simulated 4g
|Simulated 4g
|
|Root
|
|8.1.0
|Running Chrome looking CPU bottlenecks https://phabricator.wikimedia.org/T250686
|First views and second views in user journey
|-
|-
|ZY2242HHQ8
|ZY322H9XKL
|Moto G4
|Motorola Moto G5 #3
|Simulated 3g
|Simulated 4g
|Root
|Root
|7.1.1
|8.1.0
|Running mobile and user journeys tests
|
|-
|-
|ZY3222N2CZ
|R58NC31FK3Y
|Moto G5
|Samsung A51
|Traffic back to the Mac Mini
|WebPageReplay
|Root
|Root
|7.0
|10
|Running against WebPageReplay
|Running against WebPageReplay
|-
|-
|ZY322HX8RC
|
|Moto G5
|Apple iPhone 11
|Simulated 3g
|Simulated 3g
|
|
|8.1.0
|15.2.1
|Running tests using Firefox
|-
|ZY223LNPVM
|Moto Z
|Simulated 3g
|
|
|8.0.0
|Used for testing out new things
|}
|}
Using rooted phones makes it possible to stabilize CPU and GPU performance by configuring governors.
Using rooted phones makes it possible to stabilise CPU and GPU performance by configuring governors.
 
=== BitBar setup ===
At BitBar our test use a generic setup with a start bash file (''run-tests.sh'') and a ''secrets.json'' file that are uploaded in the BitBar GUI in a zip file. The bash file is called when a test is started and looks like this:<syntaxhighlight lang="bash">
# We unpack tests.zip that contains the secrets.json configuration file
unzip tests.zip
 
# Clone the git repo where we have all the code
git clone https://github.com/wikimedia/performance-mobile-synthetic-monitoring-tests.git
cd performance-mobile-synthetic-monitoring-tests
 
# There's a hack on BitBar where you can pass on a parameter
echo "The first is $1"
./start.sh $1
 
</syntaxhighlight>And the ''secrets.json'' file contains configuration to sitespeed.io to be able to send metrics to our Graphite instance and send data to S3. All tests then extends that configuration file and we can have those configurations file open to the public in our Git repo.
 
We then have cronjob calling the API using CURL with settings to use our zip file and sending one extra parameter that will choose what test to run.


== Performance tests ==
== Performance tests ==

Revision as of 10:16, 28 September 2022

Summary

In the path to get more realistic performance metric run tests on real mobile devices. That makes it easier to find performance regressions. We use BitBar as the provider of our devices. All tasks are reported under the Performance Device Lab tag in Phabricator.

Performance testing on mobile phones

Running tests on mobile phones we want a stable environment that do not change, so we can see performance regressions. To make that happen we use:

  • A stable network: We throttle the connection to look like a 3g or 4g connection.By limiting the upload/download speed and adding delay, we make the phone get requests in a more realistic scenario. By making sure the network is the same all the time, the network will not affect our metrics.
  • Low phone temperature: We measure the battery temperature as a proxy for CPU temperature. Android phones change behavior when the CPU gets warm and we want to avoid that. Some of our phones are rooted to try to make sure the phone has the same performance characteristics. We use the same settings as the Mozilla performance team to setup the rooted phones. We measure the temperature before and after we start a test. If the temperature is too high we wait X minutes and try again.

Setup

We use five mobile phones, a server and two wifi:s setup with throttled connection to simulate 3g and 4g traffic. The wifi connections is provided by two Raspberry Pis 4 running humble.

Setup showing the mobile performance device lab.

The workflow: The jobs are started on the server that runs sitespeed.io that drives the phones using WebDriver. The configuration and URLs to tests exists in a public Git repo. The tests runs on the phones, access Wikipedia and we record a video of the screen and analyze the result to get visual metrics. The metrics is sent to our Graphite instance and the test results (screenshots, HTML result pages) are stored on S3. We also run one tests using WebPageReplay where we record and replay Wikipedia locally on the server to try to get as stable metrics as possible between runs.

File:Performance-device-lab-at-bitbar.jpg
Performance device lab at BitBar.

Setup the phones

BitBar is handling the setup of phones. If we need to change anything we need to contact them.

We have five phones running at the Performance Device Lab with the following setup.

Id Type Internet connection Extras OS version Usage
ZY322DJBW9 Motorola Moto G5 #1 Simulated 3g Root 8.1.0 User journey: login
ZY322GXQN8 Motorola Moto G5 #2 Simulated 4g Root 8.1.0 First views and second views in user journey
ZY322H9XKL Motorola Moto G5 #3 Simulated 4g Root 8.1.0
R58NC31FK3Y Samsung A51 WebPageReplay Root 10 Running against WebPageReplay
Apple iPhone 11 Simulated 3g 15.2.1

Using rooted phones makes it possible to stabilise CPU and GPU performance by configuring governors.

BitBar setup

At BitBar our test use a generic setup with a start bash file (run-tests.sh) and a secrets.json file that are uploaded in the BitBar GUI in a zip file. The bash file is called when a test is started and looks like this:

# We unpack tests.zip that contains the secrets.json configuration file
unzip tests.zip

# Clone the git repo where we have all the code
git clone https://github.com/wikimedia/performance-mobile-synthetic-monitoring-tests.git
cd performance-mobile-synthetic-monitoring-tests

# There's a hack on BitBar where you can pass on a parameter
echo "The first is $1"
./start.sh $1

And the secrets.json file contains configuration to sitespeed.io to be able to send metrics to our Graphite instance and send data to S3. All tests then extends that configuration file and we can have those configurations file open to the public in our Git repo.

We then have cronjob calling the API using CURL with settings to use our zip file and sending one extra parameter that will choose what test to run.

Performance tests

All configuration and setups for our tests lives in Gerrit in performance/mobile-synthetic-monitoring-tests. To add or change tests you clone the repo and send in your change for review.

Add a test

All configuration files exists in our synthetic monitoring tests repo. Clone the repo and go into the tests folder:

git clone ssh://USERNAME@gerrit.wikimedia.org:29418/performance/mobile-synthetic-monitoring-tests.git

cd mobile-synthetic-monitoring-tests/tests

Change configuration

Troubleshooting

The device is offline

If the device is offline you can see that in the log, look for The phone state is offline. You can also verify that by running adb devices. Talk with the Wikimedia Performance team and we will contact Kobiton.

Missing data for a phone

If you see in the graphs in Grafana that data is missing you need to check the logs on the server. Each phone has their own log. Say that the phone ZY322HX8RC-Moto-G5 has the problem, then go to the directory performance-mobile-synthetic-monitoring-tests/logs and check the log file named ZY322HX8RC-Moto-G5.log. Look for error messages in the log file.

Dashboards

We have three dashboards:

Outstanding issues

There are a couple of issues that we need to fix in the current setup