Docker-registry

We run our own Docker registry at docker-registry.wikimedia.org . Internally the domain docker-registry.discovery.wmnet is also used. The registry is used by our k8s cluster, CI, and local development.

It is highly available ( docker_registry_ha Puppet module) and backed by Swift . Although we run it active/passive because of the swift replication lag.

The docker-registry nodes consist of the docker registry itself as well as an nginx reverse-proxy in front to handle authentication as well as local caching .

Browsing

Visit https://docker-registry.wikimedia.org/ to see a list of images and their tags. The listing is updated on a hourly timer and is done by the registry-homepage-builder.py script in Puppet.

Downloading images

Despite the name, the docker-registry is usable by any OCI container tool, including podman. Nearly all images may be publicly downloaded, examined, run, etc. The only exception is images under the restricted/ namespace, which contain non-disclosed security patches and require specific credentials to fetch.

Kubernetes nodes use Dragonfly to pull images.

Uploading images

For services we recommend using the Deployment pipeline which is Blubber .

For other docker images, like infrastructure images, we manage them using docker-pkg, see: Kubernetes/Images#Image_building

Hosts that want to upload images must be individually listed in Puppet hiera.

Access control

The upstream docker-registry software provides no access control, so it is implemented at the nginx level, which restricts GET/POST/etc. requests accordingly. As of 2021-03-18, the following accounts exist:

ci-restricted : Can pull and push any image (including "restricted/"). Used by releases servers that build the restricted MediaWiki production image.
ci-build : Can pull and push any non-restricted image. Used by contint servers via docker-pkg and the deployment pipeline.
prod-build : Can pull and push any non-restricted image. Used by build2001.codfw.wmnet via docker-pkg and build-base-images.
kubernetes : Can pull any image (including "restricted/"). Used by k8s nodes to pull images, including the restricted MediaWiki production image.
- See Kubernetes/Clusters/New#Access to restricted docker images for more details.

The passwords are all deployed using the private puppet repo. In case rotation is needed (e.g. compromise), grepping for <name>_user_password should find all uses (switch hyphens to underscores).

jwt-authorizer

docker-registry also supports authorization using JSON Web Tokens. A dedicated daemon is running which handles jwt validation. See Docker-registry/jwt-authorizer for more information.

Deleting images

To delete an image entirely, you may use the tool docker-registryctl on the current build host. It will do it's best to remove the tags/image from the registry, despite the circumstances . Note: the domain used here is important. discovery.wmnet has to be used, so you will have to adjust it if you copy and paste from the browser UI.

elukey@build2001:~$ sudo -i docker-registryctl delete-tags docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services
We're about to delete the following tags for image docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:
2021-07-28-175322-production
stable
Ok to proceed? (y/n)y
docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:2021-07-28-175322-production[DONE]
docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:stable[GONE]

Maintenance

The docker-registry nodes consist of the docker registry itself as well as an nginx reverse-proxy in front to handle authentication as well as local caching .

Service management

The following units run on the registry nodes:

Service unit	Description
`docker-registry-ha-jwt.service`	JSON Web Token authoriser
`docker-registry-swift.service`	Registry instance backed by Swift
`docker-registry-ml.service`	Registry instance serving the ML/LiftWing namespace
`docker-registry-restricted.service`	Registry instance serving restricted images

Restarting

Before restarting any service, confirm which node is currently active and prefer restarting the passive node where possible to avoid unnecessary disruption.

cumin1003:$ sudo  confctl --object-type  discovery select 'dnsdisc=docker-registry' get

sudo systemctl restart docker-registry-ha-jwt.service
sudo systemctl restart docker-registry-swift.service
sudo systemctl restart docker-registry-ml.service
sudo systemctl restart docker-registry-restricted.service

To check status:

sudo systemctl status docker-registry-ha-jwt.service

Note that nginx sits in front of all registry instances and handles authentication and local caching.

sudo systemctl status nginx.service
sudo systemctl restart nginx.service

Post-restart verification - httpbb

Run the httpbb tests against the restarted node to confirm it is serving correctly. Make sure the instance is not in read-only mode first ( profile::docker_registry::read_only_mode: false ):

sudo httpbb /srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml --hosts 'registry2004.codfw.wmnet'