You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Kubernetes/Kubernetes Workshop/Load Testing

From Wikitech-static
< Kubernetes‎ | Kubernetes Workshop
Revision as of 06:25, 11 March 2021 by imported>Wolfgang Kandek (Wolfgang Kandek moved page Kubernetes/Kubernetes Workshop Load Testing to Kubernetes/Kubernetes Workshop/Load Testing)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

The basic idea is to use a program like apachebench (in package apache2-utils) or hey (https://github.com/rakyll/hey) to generate load on the service. Both of these programs retrieve the same URL multiple times with a concurrency factor, say 10,000 times total with a concurrency of 10 and then report back requests per second, errors, latencies and latency histograms. We will use the calculator-service demo service for this walk through.

Minikube normally loads docker images from dockerhub, or any other repository but can use a local image, if the policy is set to Never load. We can build a local docker image for the calculator-service for easier manipulation and faster updates:

  • check out the calculator-service repo from gerrit or github
  • eval $(minikube docker-env)
  • docker images # to check
  • docker build --tag wmfcalc-mk3 .

The calculator-service is simple and should not present major CPU usage. Let's start with a low CPU allocation in kubernetes: CPU = 100m. calculator-service reports its memory footprint on its /metrics page and it is fairly low (<14 MB). We can set memory = 16 Mb. Both of these variables are set in the deployment file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: calc
  labels:
    app: calc
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: calc
  template:
    metadata:
      labels:
        app: calc
    spec:
      containers:
       - name: calc
         image: docker-registry.wikimedia.org/wikimedia/blubber-doc-example-calculator-service:stable
         imagePullPolicy: Always
         resources:
           requests:
             memory: "16Mi"
             cpu: "100m"
           limits:
             memory: "16Mi"
             cpu: "100m"

We start with 1 replica for a baseline, note the requests per second, then increase the number of replicas and see if and how the service scales.

Start the service and deployment.

Then run tests with ab - simple math expression, then more complicated, one with a parsing error, and one that induces the length error.

Useful URLs:

  • curl http://192.168.49.2:32459/api?2+3 # to test
  • curl http://192.168.49.2:32459/metrics # to get metrics
  • curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) # more complex
  • curl http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3)) # syntax error
  • curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa # length error input is limited to 60 characters
  • ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+3 | grep Requests Requests per second:    168.51 [#/sec] (mean)
  • curl http://192.168.49.2:32459/metrics # to get metrics

Summary: about 80 rps - memory is at 13 MB - see the Appendix for more data

Now increase the number of replicas to 2, 4, 8, then 16. This results in a plateau at 1000 rps with ab,  but hey binary seems to get more request up to the 1900 rps area

Overall a configuration with 2 replicas using 100m CPU and 16 Mb memory will be able to serve 150+ rps and have some high availability and is a good baseline for our first release.

Appendix:

One Replica

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    129.23 [#/sec] (mean)

ab -q -n 1000 -c 2 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    159.30 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    42.55 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    164.13 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    138.44 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    168.51 [#/sec] (mean)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))
{"operation":"2+38+(44)(64)((66555/2-3))","result":"7031834.0"}

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    93.77 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    106.45 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    104.89 [#/sec] (mean)

## Parse error (^ is not supported)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3))
{"operation":"2+38+(44)(64)((66555/2-3))","result":"None"}

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)*((66555/2-3)) | grep Requests
Requests per second:    86.60 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3)) | grep Requests
Requests per second:    95.75 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3)) | grep Requests
Requests per second:    95.08 [#/sec] (mean)

## Length error (input is limited to 60 characters)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa| grep Requests
Requests per second:    195.40 [#/sec] (mean)

curl http://192.168.49.2:32459/metrics
start_time 1613141183.1434238
wellformed_total{method="get"} 1000
wellformed_total{method="post"} 0
nonwellformed_total 2000
memory 13492224
duration_bucket{le="1"} 0
duration_bucket{le="2"} 0
duration_bucket{le="4"} 994
duration_bucket{le="8"} 209
duration_bucket{le="16"} 628
duration_bucket{le="32"} 40
duration_bucket{le="32+"} 129

## 4 replicas

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    410.19 [#/sec] (mean)

## 8 replicas

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    954.36 [#/sec] (mean)

## 16 replicas - 1st run too fast 2000+ rps seemed suspicious - increase number of  requests

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    2173.27 [#/sec] (mean)

ab -q -n 10000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1132.95 [#/sec] (mean)

ab -q -n 20000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1011.18 [#/sec] (mean)

## 24 replicas

ab -q -n 20000 -c 16 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1272.75 [#/sec] (mean)

hey -n 20000 -c 16 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
 Requests/sec: 1941.3902