Services/Deployment

Regular Deployment

There are a lot of moving parts in our production stack -- MediaWiki, its extensions, various back-end services, HTTPS handlers, caches, just to name a few. It is thus important that you communicate your deployment schedules on the Deployments page.

The deployment process starts with updating the deploy repository. Go into your source repository and update it with:

$ ./server.js build --deploy-repo --force --review

The build script will update the pointer of the deploy repository's submodule, create a Docker container in which it will install the module dependencies and send the changes to Gerrit. Review them and merge. Next, log onto `deployment.eqiad.wmnet` and update the repo there:

$ cd /srv/deployment/<service-name>/deploy
$ git pull && git submodule update --init

In the #wikimedia-operations IRC channel announce the deployment by logging it into the Server Admin Log with !log <service-name> deploying <deploy-repo-sha1>. Now, proceed to do the dpeloyment from deployment.eqiad.wmnet:

$ deploy

Scap3 will deploy the code, restart the service and check its port and health. In case it detects some problems on the canary node, it will suggest to perform a roll-back. Otherwise it will proceed to deploying it to the rest of the nodes, which completes the deployment process.

Dealing with Problems

Deployment Debugging

Scap3 includes a utility which can be used to monitor the output of the commands executed on the target nodes. Fire up a second terminal, connect to deployment.eqiad.wmnet and execute the deploy-log command from /srv/deployment/<service-name>/deploy before starting the deployment. The output should help you figure out what went wrong.

If you haven't started an instance of deploy-log during the deploy, but it went badly, you can still recuperate the logs by running deploy-log --latest.

Reverting a Deployment

Sometimes the deployment process goes well, but the code that was deployed isn't functioning properly. To revert a deployment and bring the code on the target nodes to a previous state, find the deploy repository's SHA1 that contained the good code and then deploy it with:

$ deploy --rev <sha1>