You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

RESTBase

From Wikitech-static
Revision as of 14:49, 22 June 2015 by imported>GWicke (→‎Staging deploy)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

RESTBase is an API proxy serving the REST API at /api/rest_v1/. It uses Cassandra as a storage backend.

It is currently running on restbase100{1..6}.eqiad.wmnet, and shares the hardware with Cassandra instances.

Deployment and config changes

Preliminaries: Get the Ansible deploy scripts

We are using a set of simple Ansible deploy scripts to coordinate rolling deploys and restarts. These are currently not installed on a deploy host (FIXME!), so you need to check them out locally:

git clone https://github.com/wikimedia/ansible-deploy.git

The scripts assume that you have a working SSH proxy command setup, so that ssh restbase1001.eqiad works. The following ansible commands are assumed to be executed from within the ansible-deploy checkout (so cd ansible-deploy).

Preparing the deploy repository

RESTBase is a service-runner based application, to prepare the software repository for deploy, follow the instructions on updating, here.

Deploy to staging

Before deploying to production, we generally deploy to the staging cluster (xenon, praseodymium and cerium) first. We deploy via Ansible, which handles the full rolling deploy, including restarts and checks.

In the ansible tree: ansible-playbook -i staging roles/restbase/deploy.yml

Tip: You can also limit the deploy to some hosts only: ansible-playbook -i staging -l xenon.* roles/restbase/deploy.yml. Regexps are also supported, which is especially useful for numbered hosts in production: -l ~restbase.100[1-2].*

Deploy to production

If things went well in staging, then you can proceed to deploy to production.

In the ansible tree: ansible-playbook -i production roles/restbase/deploy.yml

Roll-back a deploy

Modify the restbase version in group_vars/restbase from 'master' to the revision you'd like to roll back to. Then, deploy as usual:

In the ansible tree: ansible-playbook -i production roles/restbase/deploy.yml

Rolling restart

In the ansible tree: ansible-playbook -i production roles/restbase/restart.yml

Deploy config changes

As config changes can trigger database changes in RESTBase, it is very important that those are deployed in a rolling fashion as well. The configuration templating is handled by puppet, which doesn't directly support rolling deploys. To work around this, we need to manually perform a rolling deploy by disabling puppet & then re-enabling it one by one. Procedure (note: all of the following commands need to be run as root):

  • Disable puppet on all restbase* hosts, to make sure that config changes are applied one host at a time: puppet agent --disable
  • For each node:
    • re-enable / run puppet: puppet agent --enable; puppet agent -tv
    • restart restbase with systemctl restart restbase
    • verify that RB is back up with curl http://<boxip>:7231/

TODO: Integrate with safe rolling restarts above

After each deploy