Istio
From the upstream website : Istio extends Kubernetes to establish a programmable, application-aware network using the powerful Envoy service proxy. Working with both Kubernetes and traditional workloads, Istio brings standard, universal traffic management, telemetry, and security to complex deployments.
Istio binary sources
In order to deploy Istio to our Kubernetes clusters, we use three main sources for the binaries needed:
- istioctl Debian package (repo https://gerrit.wikimedia.org/r/admin/repos/operations/debs/istioctl )
- istio-cni Debian package (repo https://gerrit.wikimedia.org/r/admin/repos/operations/debs/istio )
- The istio-build Docker image (repo https://gerrit.wikimedia.org/r/admin/repos/operations/docker-images/production-images , docker-registry https://docker-registry.wikimedia.org/istio/build/tags/ )
Let's review each source to understand why we need it and how it is used (at a very high level).
Istioctl
The istioctl binary is a Go-based tool shipped by upstream that replaces the need for a dedicated Istio Helm chart. We use it to deploy CRDs and k8s settings, bypassing helmfile, but keeping the configuration yamls in the same repo (see https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/custom_deploy.d/istio/ ). We chose to follow what upstream recommends, since upgrades and general maintenance is easier with istioctl.
The tool is able to perform its job because, behind the scenes, it embeds helm3 and several charts that are maintained by upstream. Every version of istio ships with its own istioctl, so we needed a way to have multiple versions of istioctl on the deployment node (to support the fact that not all kubernetes nodes will run with the same Istio version at any given time). We created a Debian package that collects multiple releases of Istio from Github, and that packages istioctl binaries named istioctl-#version (for example, istioctl-195).
Packaging
We don't actually build istioctl but package it from upstream binary releases as described above.
Because of that, you will need to set HTTP proxy variables for internet access on the build host.
The general process to follow is:
- Check out operations/debs/istioctl on your workstation
- Create a patch to bump the debian changelog
Double-check that the environment variables DEBEMAIL and DEBFULLNAME are set correctly before running debchange (dch).
export NEW_VERSION=1.29.4 # New istioctl version you want to package
dch -v ${NEW_VERSION}-1 -D bookworm-wikimedia "Update to v${NEW_VERSION}"
git commit debian/changelog
# Update debign/rules and debian/istioctl.bash-completion with the new version
# and drop the oldest version if applicable
git review
- Merge
- Check out operations/debs/istioctl on the build host:
git clone "https://gerrit.wikimedia.org/r/operations/debs/istioctl"
- Build the packages:
# Ensure you allow networking in pbuilder
# This option needs to be in the file, an environment variable will *not* work!
echo "USENETWORK=yes" >> ~/.pbuilderrc
# Build the package
https_proxy=http://webproxy.$(hostname -d):8080 DIST=bookworm pdebuild
istio-cni
Packaging
For packaging instructions, please see the debian/README.md in git repo.
Publishing
# On apt1002, copy the packages from the build host
# pbuilder-result is an rsync alias to /var/cache/pbuilder/result on the build host.
rsync -vaz build2001.codfw.wmnet::pbuilder-result/bookworm-amd64/istioctl*<PACKAGE VERSION>* .
# Import packages to the main apt component
sudo -i reprepro --ignore=wrongdistribution include bookworm-wikimedia /path/to/<PACKAGE>.changes
# istioctl packages can be copied between distros:
sudo -i reprepro copysrc trixie-wikimedia bookworm-wikimedia istioctl
Istio build Docker image
This is where the Istio binaries are built (proxyv2 and pilot) to then be included in the Docker images that Istio needs to operate. We mimic what upstream does with few extra settings, since our policy is to avoid using untrusted repositories like Dockerhub directly. More information about the build process in the build Dockerfile, but it is basically calling some targets of a Makefile.
The images can be build on the build host:
user@host:~$ sudo -i
root@host:~# cd /srv/images/production-images/
root@host:~# git pull
root@host:~# /srv/deployment/docker-pkg/venv/bin/docker-pkg -c /etc/production-images/config-istio.yaml build istio/
Istio-cni
In order to implement a service mesh, upstream provides a way to inject Envoy sidecars to kubernetes pod to intercept (transparently) traffic and apply routing / traffic policies over it (most useful ones are proxying and circuit breaking). The trick that Istio uses it to leverage special binaries implementing the CNI interface to inject iptables policies when the pod is being created (more info https://istio.io/latest/docs/setup/additional-setup/cni/ ).
The CNI binaries need to be shipped to the Kubernetes nodes, and we use a dedicated Debian package for it (we chose not to follow the install-cni Daemonset approach listed in the upstream page due to the security capabilities that need to be granted to the pods).