You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Kubernetes/Kubernetes Workshop/Load Testing: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Wolfgang Kandek
 
imported>Mbaoma
No edit summary
 
(2 intermediate revisions by 2 users not shown)
Line 1: Line 1:
The basic idea is to use a program like apachebench (in package apache2-utils) or hey (<nowiki>https://github.com/rakyll/hey</nowiki>) to generate load on the service. Both of these programs retrieve the same URL multiple times with a concurrency factor, say 10,000 times total with a concurrency of 10 and then report back requests per second, errors, latencies and latency histograms. We will use the calculator-service demo service for this walk through.
== Overview ==
At the end of this module, you should be able to:
* Carry out load testing using k6.io.


Minikube normally loads docker images from dockerhub, or any other repository but can use a local image, if the policy is set to Never load. We can build a local docker image for the calculator-service for easier manipulation and faster updates:
== Remote Testing with k6.io ==
This module uses a program like ''apachebench'' (in package apache2-utils) or ''[https://github.com/rakyll/hey hey]'' to generate load on a service. These programs then report back requests, errors, latencies, and latency histograms per second. You will use the calculator-service in [[Kubernetes/Kubernetes Workshop/Kubernetes at the Wikimedia Foundation (WMF)|Module 10]] for this walk-through.


* check out the calculator-service repo from gerrit or github
Minikube usually loads Docker images from Docker Hub or any other repository, but you can use a local image if you set the ''ImagePullPolicy'' to '''Never''' in the <code>values.yaml</code> file.  
* eval $(minikube docker-env)
* docker images # to check
* docker build --tag wmfcalc-mk3 .


The calculator-service is simple and should not present major CPU usage. Let's start with a low CPU allocation in kubernetes: CPU = 100m. calculator-service reports its memory footprint on its /metrics page and it is fairly low (<14 MB). We can set memory = 16 Mb. Both of these variables are set in the deployment file.<syntaxhighlight lang="yaml">
You can build a local Docker image for the calculator-service for easier manipulation and faster updates. Take the following steps:
* Clone the calculator-service repository from [https://gerrit.wikimedia.org/g/blubber-doc/example/calculator-service Gerrit].
<syntaxhighlight lang="bash">
$ minikube start
$ eval $(minikube docker-env)
$ docker build --tag wmfcalc-mk3 .
$ docker run -p 8080:8080 wmfcalc-mk3
</syntaxhighlight>
The calculator service is simple and should not present significant CPU usage. Start with a low CPU allocation in Kubernetes: CPU = 100m. calculator-service reports its memory footprint on its /metrics page, which is relatively low (<14 MB). You can set memory to 16 Mb. Set both variables in the deployment file.
 
'''deployment.yaml'''
<syntaxhighlight lang="bash">
apiVersion: apps/v1
apiVersion: apps/v1
kind: Deployment
kind: Deployment
Line 38: Line 49:
             memory: "16Mi"
             memory: "16Mi"
             cpu: "100m"
             cpu: "100m"
</syntaxhighlight>We start with 1 replica for a baseline, note the requests per second, then increase the number of replicas and see if and how the service scales.
</syntaxhighlight>
 
Create a deployment and make curl requests:
<syntaxhighlight lang="bash">
$ kubectl create -f deployment.yaml
$ curl localhost:8080/api?2+3 # to test
{"operation":"2+3","result":"5"}
$ curl http://<ip:port>/metrics # to get metrics
$ curl http://<ip:port>/api?2+38+(44)(64)((66555/2-3)) # more complex
$ curl http://<ip:port>/api?2+38+(44)(64)((66^555/2-3)) # syntax error
$ ab -q -n 1000 -c 8 http://<ip:port>/api?2+3 | grep Requests #Requests per second: 168.51 [#/sec] (mean)
$ curl http://<ip:port>/metrics
$ curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa #ength error input is limited to 60 characters
</syntaxhighlight>
 
'''Note''':
* Start with one replica for a baseline, take note of the requests per second, then increase the number of replicas to see whether the service scales.
* Start the service and deployment.
* Then run tests with ab - simple math expression, a more complicated one with a parsing error, and one that induces the length error.
 
== Assessing Baseline Metrics ==
About 80 rps - memory is at 13 MB - see the Appendix for more data. When you increase the number of replicas to 2, 4, 8, then 16, it results in a plateau at 1000 rps with ab, but the hey binary seems to get more requests up to the 1900 rps area.
 
Overall a configuration with two replicas using 100m CPU and 16 Mb memory will serve 150+ rps and have some high availability and is a good baseline for our first release.
 
== Local Testing with k6.io ==
k6.io is an external testing service. You can record test sequences with a browser or code them using Javascript. k6.io has test execution machines worldwide.


Start the service and deployment.
The service to test needs to be externally available. In this test, you will use your Kubernetes (k8s) installation from Wikimedia Cloud Service (WMCS) from [[Kubernetes/Kubernetes Workshop/Building a production ready cluster|Module 6]], but in addition, you need a floating IP to provide external access.


Then run tests with ab - simple math expression, then more complicated, one with a parsing error, and one that induces the length error.
# Take the following steps:
# Install k8s on WMCS (see chapter 6)
# Deploy calc/1.2
# Define a service using a nodeport. Find the port mapped (30000+)
# Map the floating IP to one of the nodes1
# Open the firewall for port from Step 3
# Test access from the Internet:
<syntaxhighlight lang="bash">
$ curl http://{floating ip}:{port}/api?2+6 and http://{floating ip}:{port}/metrics
</syntaxhighlight>
# Log in to k6.io. (A trial account would suffice)
# Run a script to test calc. Here is an example created from k6.io's library. The test is as follows: a  5-minute test, 1 minute ramping up to 20 Virtual users, then 3 minutes at 20 Virtual Users, then ramping down. The test is a simple get, followed by a 1-second sleep call.
<syntaxhighlight lang="bash">
import { sleep } from 'k6'
import http from 'k6/http'


Useful URLs:
// See https://k6.io/docs/using-k6/options
export const options = {
  stages: [
    { duration: '1m', target: 20 },
    { duration: '3m', target: 20 },
    { duration: '1m', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.02'], // http errors should be less than 2%
    http_req_duration: ['p(95)<2000'], // 95% requests should be below 2s
  },
  ext: {
    loadimpact: {
      distribution: {
        'amazon:us:ashburn': { loadZone: 'amazon:us:ashburn', percent: 100 },
      },
    },
  },
}


* curl <nowiki>http://192.168.49.2:32459/api?2+3</nowiki> # to test
export default function main() {
* curl <nowiki>http://192.168.49.2:32459/metrics</nowiki> # to get metrics
  let response = http.get('http://185.15.56.95:30162/api?2+3')
* curl <nowiki>http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))</nowiki> # more complex
  sleep(1)
* curl <nowiki>http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3))</nowiki> # syntax error
}
</syntaxhighlight>


* curl <nowiki>http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa</nowiki> # length error input is limited to 60 characters
[[File:K8s Test results.png|thumb|center]]
* ab -q -n 1000 -c 8 <nowiki>http://192.168.49.2:32459/api?2+3</nowiki> | grep Requests  Requests per second:    168.51 [#/sec] (mean)
* curl <nowiki>http://192.168.49.2:32459/metrics</nowiki> # to get metrics


Summary: about 80 rps - memory is at 13 MB - see the Appendix for more data
== Output Analysis ==
* Output from calc's metrics collection:
<syntaxhighlight lang="bash">
$ curl http://185.15.56.95:30162/metrics
start_time 1618522447.9971263
wellformed_total{method="get"} 23669
wellformed_total{method="post"} 0
nonwellformed_total 0
memory 18292736
duration_bucket{le="1"} 6
duration_bucket{le="2"} 562
duration_bucket{le="4"} 11376
duration_bucket{le="8"} 11488
duration_bucket{le="16"} 200
duration_bucket{le="32"} 27
duration_bucket{le="32+"} 13
</syntaxhighlight>


Now increase the number of replicas to 2, 4, 8, then 16. This results in a plateau at 1000 rps with ab,  but hey binary seems to get more request up to the 1900 rps area
* More test runs:
50 users from São Paulo, Brazil
[[File:50 users from São Paulo.png|thumb|center]]


Overall a configuration with 2 replicas using 100m CPU and 16 Mb memory will be able to serve 150+ rps and have some high availability and is a good baseline for our first release.
50 users from Mumbai, India
[[File:50 users from Mumbai.png|thumb|center]]


=== Appendix: ===
=== Appendix: ===

Latest revision as of 10:03, 17 June 2022

Overview

At the end of this module, you should be able to:

  • Carry out load testing using k6.io.

Remote Testing with k6.io

This module uses a program like apachebench (in package apache2-utils) or hey to generate load on a service. These programs then report back requests, errors, latencies, and latency histograms per second. You will use the calculator-service in Module 10 for this walk-through.

Minikube usually loads Docker images from Docker Hub or any other repository, but you can use a local image if you set the ImagePullPolicy to Never in the values.yaml file.

You can build a local Docker image for the calculator-service for easier manipulation and faster updates. Take the following steps:

  • Clone the calculator-service repository from Gerrit.
$ minikube start
$ eval $(minikube docker-env)
$ docker build --tag wmfcalc-mk3 .
$ docker run -p 8080:8080 wmfcalc-mk3

The calculator service is simple and should not present significant CPU usage. Start with a low CPU allocation in Kubernetes: CPU = 100m. calculator-service reports its memory footprint on its /metrics page, which is relatively low (<14 MB). You can set memory to 16 Mb. Set both variables in the deployment file.

deployment.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: calc
  labels:
    app: calc
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: calc
  template:
    metadata:
      labels:
        app: calc
    spec:
      containers:
       - name: calc
         image: docker-registry.wikimedia.org/wikimedia/blubber-doc-example-calculator-service:stable
         imagePullPolicy: Always
         resources:
           requests:
             memory: "16Mi"
             cpu: "100m"
           limits:
             memory: "16Mi"
             cpu: "100m"

Create a deployment and make curl requests: 

$ kubectl create -f deployment.yaml
$ curl localhost:8080/api?2+3 # to test
{"operation":"2+3","result":"5"}
$ curl http://<ip:port>/metrics # to get metrics
$ curl http://<ip:port>/api?2+38+(44)(64)((66555/2-3)) # more complex
$ curl http://<ip:port>/api?2+38+(44)(64)((66^555/2-3)) # syntax error
$ ab -q -n 1000 -c 8 http://<ip:port>/api?2+3 | grep Requests #Requests per second: 168.51 [#/sec] (mean)
$ curl http://<ip:port>/metrics
$ curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa #ength error input is limited to 60 characters

Note:

  • Start with one replica for a baseline, take note of the requests per second, then increase the number of replicas to see whether the service scales.
  • Start the service and deployment.
  • Then run tests with ab - simple math expression, a more complicated one with a parsing error, and one that induces the length error.

Assessing Baseline Metrics

About 80 rps - memory is at 13 MB - see the Appendix for more data. When you increase the number of replicas to 2, 4, 8, then 16, it results in a plateau at 1000 rps with ab, but the hey binary seems to get more requests up to the 1900 rps area.

Overall a configuration with two replicas using 100m CPU and 16 Mb memory will serve 150+ rps and have some high availability and is a good baseline for our first release.

Local Testing with k6.io

k6.io is an external testing service. You can record test sequences with a browser or code them using Javascript. k6.io has test execution machines worldwide.

The service to test needs to be externally available. In this test, you will use your Kubernetes (k8s) installation from Wikimedia Cloud Service (WMCS) from Module 6, but in addition, you need a floating IP to provide external access.

  1. Take the following steps:
  2. Install k8s on WMCS (see chapter 6)
  3. Deploy calc/1.2
  4. Define a service using a nodeport. Find the port mapped (30000+)
  5. Map the floating IP to one of the nodes1
  6. Open the firewall for port from Step 3
  7. Test access from the Internet:
$ curl http://{floating ip}:{port}/api?2+6 and http://{floating ip}:{port}/metrics
  1. Log in to k6.io. (A trial account would suffice)
  2. Run a script to test calc. Here is an example created from k6.io's library. The test is as follows: a 5-minute test, 1 minute ramping up to 20 Virtual users, then 3 minutes at 20 Virtual Users, then ramping down. The test is a simple get, followed by a 1-second sleep call.
import { sleep } from 'k6'
import http from 'k6/http'

// See https://k6.io/docs/using-k6/options
export const options = {
  stages: [
    { duration: '1m', target: 20 },
    { duration: '3m', target: 20 },
    { duration: '1m', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.02'], // http errors should be less than 2%
    http_req_duration: ['p(95)<2000'], // 95% requests should be below 2s
  },
  ext: {
    loadimpact: {
      distribution: {
        'amazon:us:ashburn': { loadZone: 'amazon:us:ashburn', percent: 100 },
      },
    },
  },
}

export default function main() {
  let response = http.get('http://185.15.56.95:30162/api?2+3')
  sleep(1)
}
File:K8s Test results.png

Output Analysis

  • Output from calc's metrics collection:
$ curl http://185.15.56.95:30162/metrics
start_time 1618522447.9971263
wellformed_total{method="get"} 23669
wellformed_total{method="post"} 0
nonwellformed_total 0
memory 18292736
duration_bucket{le="1"} 6
duration_bucket{le="2"} 562
duration_bucket{le="4"} 11376
duration_bucket{le="8"} 11488
duration_bucket{le="16"} 200
duration_bucket{le="32"} 27
duration_bucket{le="32+"} 13
  • More test runs:

50 users from São Paulo, Brazil

File:50 users from São Paulo.png

50 users from Mumbai, India

File:50 users from Mumbai.png

Appendix:

One Replica

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    129.23 [#/sec] (mean)

ab -q -n 1000 -c 2 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    159.30 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    42.55 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    164.13 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    138.44 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    168.51 [#/sec] (mean)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))
{"operation":"2+38+(44)(64)((66555/2-3))","result":"7031834.0"}

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    93.77 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    106.45 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    104.89 [#/sec] (mean)

## Parse error (^ is not supported)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3))
{"operation":"2+38+(44)(64)((66555/2-3))","result":"None"}

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)*((66555/2-3)) | grep Requests
Requests per second:    86.60 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3)) | grep Requests
Requests per second:    95.75 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3)) | grep Requests
Requests per second:    95.08 [#/sec] (mean)

## Length error (input is limited to 60 characters)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa| grep Requests
Requests per second:    195.40 [#/sec] (mean)

curl http://192.168.49.2:32459/metrics
start_time 1613141183.1434238
wellformed_total{method="get"} 1000
wellformed_total{method="post"} 0
nonwellformed_total 2000
memory 13492224
duration_bucket{le="1"} 0
duration_bucket{le="2"} 0
duration_bucket{le="4"} 994
duration_bucket{le="8"} 209
duration_bucket{le="16"} 628
duration_bucket{le="32"} 40
duration_bucket{le="32+"} 129

## 4 replicas

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    410.19 [#/sec] (mean)

## 8 replicas

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    954.36 [#/sec] (mean)

## 16 replicas - 1st run too fast 2000+ rps seemed suspicious - increase number of  requests

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    2173.27 [#/sec] (mean)

ab -q -n 10000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1132.95 [#/sec] (mean)

ab -q -n 20000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1011.18 [#/sec] (mean)

## 24 replicas

ab -q -n 20000 -c 16 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1272.75 [#/sec] (mean)

hey -n 20000 -c 16 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
 Requests/sec: 1941.3902
Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Kubernetes/Kubernetes_Workshop/Load_Testing&oldid=591468"