You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Services/Deployment: Difference between revisions
imported>Mobrovac mNo edit summary |
imported>BryanDavis (spelling) |
||
| Line 9: | Line 9: | ||
</pre> | </pre> | ||
The build script will update the pointer of the deploy repository's submodule, create a Docker container in which it will install the module dependencies and send the changes to Gerrit. Review them and merge. Next, log onto | The build script will update the pointer of the deploy repository's submodule, create a Docker container in which it will install the module dependencies and send the changes to Gerrit. Review them and merge. Next, log onto [[deployment.eqiad.wmnet]] and update the repo there: | ||
<pre> | <pre> | ||
| Line 16: | Line 16: | ||
</pre> | </pre> | ||
In the [irc://irc.freenode.net/wikimedia-operations <tt>#wikimedia-operations</tt>] IRC channel announce the deployment by logging it into the [[Server Admin Log]] with <code>!log <service-name> deploying <deploy-repo-sha1></code>. Now, proceed to do the | In the [irc://irc.freenode.net/wikimedia-operations <tt>#wikimedia-operations</tt>] IRC channel announce the deployment by logging it into the [[Server Admin Log]] with <code>!log <service-name> deploying <deploy-repo-sha1></code>. Now, proceed to do the deployment from <tt>deployment.eqiad.wmnet</tt>: | ||
<pre> | <pre> | ||
Revision as of 05:16, 22 April 2016
Regular Deployment
There are a lot of moving parts in our production stack -- MediaWiki, its extensions, various back-end services, HTTPS handlers, caches, just to name a few. It is thus important that you communicate your deployment schedules on the Deployments page.
The deployment process starts with updating the deploy repository. Go into your source repository and update it with:
$ ./server.js build --deploy-repo --force --review
The build script will update the pointer of the deploy repository's submodule, create a Docker container in which it will install the module dependencies and send the changes to Gerrit. Review them and merge. Next, log onto deployment.eqiad.wmnet and update the repo there:
$ cd /srv/deployment/<service-name>/deploy $ git pull && git submodule update --init
In the #wikimedia-operations IRC channel announce the deployment by logging it into the Server Admin Log with !log <service-name> deploying <deploy-repo-sha1>. Now, proceed to do the deployment from deployment.eqiad.wmnet:
$ deploy
Scap3 will deploy the code, restart the service and check its port and health. In case it detects some problems on the canary node, it will suggest to perform a roll-back. Otherwise it will proceed to deploying it to the rest of the nodes, which completes the deployment process.
Dealing with Problems
Deployment Debugging
Scap3 includes a utility which can be used to monitor the output of the commands executed on the target nodes. Fire up a second terminal, connect to deployment.eqiad.wmnet and execute the deploy-log command from /srv/deployment/<service-name>/deploy before starting the deployment. The output should help you figure out what went wrong.
If you haven't started an instance of deploy-log during the deploy, but it went badly, you can still recuperate the logs by running deploy-log --latest.
Reverting a Deployment
Sometimes the deployment process goes well, but the code that was deployed isn't functioning properly. To revert a deployment and bring the code on the target nodes to a previous state, find the deploy repository's SHA1 that contained the good code and then deploy it with:
$ deploy --rev <sha1>