You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

User:Kosta Harlan/Add Link Deployment Charts Notes

From Wikitech-static
< User:Kosta Harlan
Revision as of 10:48, 15 February 2021 by imported>Kosta Harlan (→‎Debugging)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Notes on testing out a patch for the link recommendation service with operations/deployment-charts. (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/660394)

Local charts

I used local-charts (https://gerrit.wikimedia.org/g/releng/local-charts). You could maybe avoid using this tool and use helm + kubectl directly, but using local-charts probably simplifies setting up the link recommendation service (python app + mariadb).

Note that you need to make sure you install helm version 2, not helm version 3. (Until recently the script for installing prerequisites installed v3.)

I ran into problems with Minikube so I used the Kubernetes support in the Docker for Mac application. YMMV. The instructions below assume Docker for Mac with its Kubernetes support switched on.

Build your image

The patch I wanted to test adds a cron job to load datasets. The job calls the `load-datasets.py` script in the research/mwaddlink repo. Because the patch which introduces load-datasets.py is not merged and built as a docker image I can already pull, I need to build that image on my laptop so its available for use in my laptop's kubernetes instance.

Tournesol.png Checkout patch and build research/mwaddlink image

Setup local-charts

  • Clone the local-charts repo, if you haven't already.
  • Clone the operations/deployment-charts repo

Take note of the relative paths of these two repositories, as local-charts is going to need to access files in the operations/deployment-charts repo. On my laptop, I have them cloned to these two paths:

  • ~/src/operations/deployment-charts
  • ~/src/releng/local-charts

Now in the local-charts repo, check out the link recommendation service example patch (git review -d https://gerrit.wikimedia.org/r/c/releng/local-charts/+/639347)

Then, in the operations/deployment-charts repo, checkout the cron job patch (git review -d https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/660394)

Deploy chart

Now, from the local-charts repo, run:

  • helm init
  • helm dependency update ./helm
  • helm install --name "default" -f "values.example.yaml" ./helm

That's it. The cronjob should now be deployed.

Note that I modified the values in cronjob.yaml from `@hourly` to run instead every minute, so I could get faster feedback:


spec:
  # schedule: @hourly
  schedule: "*/1 * * * *"

Note that by specifying "every minute", the first container creation attempts will fail because the mariadb service is not yet initialized. But the third or fourth container creation attempt should be successful.

If you want to make changes to the cronjob spec in the operations/deployment-charts repo, you make your changes to the cronjob.yaml file, and then back from the local-charts repo run:

  • helm del --purge default
  • helm dependency update ./helm
  • helm install --name "default" -f "values.example.yaml" ./helm

Debugging

If you want to see what is going on, you can run `kubectl get pods`:

Tournesol.png kubectl get pods

You can also look at just the cronjob:

Tournesol.png kubectl get cronjob --watch

Once a cronjob is running, you can take its ID and inspect it with `kubectl describe cronjob/linkrecommendation-default-load-datasets-{id}`.

You can also view its logs to figure out why it's not working: `kubectl logs -f linkrecommendation-default-load-datasets-{id}`.

Tournesol.png kubectl get jobs

Finally, you can view the application by using kubectl port-forward.

Tournesol.png port-forward