You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Swift Ring Management
Swift rings are managed by an automated process; you edit a YAML file that describes the hosts in the cluster, and then the
swift_ring_manager process (run by a systemd timer) gradually adjusts the deployed rings to match the state specified in the YAML file. The aim is that the YAML file should be reasonably intuitive; some common tasks are outlined here.
The swift ring manager code is installed on one front-end per Swift cluster (see the
ring_manager entry in
swift_clusters in hiera); the [code repository] is checked out into
/srv/deployment/swift_ring_manager, and the script
/usr/local/bin/swift_ring_manager is templated by puppet to ensure the correct version of python is used. That repository contains a test suite, which contains [an extensive set of example changes].
The configuration file is deployed as
/etc/swift/hosts.yaml by Puppet; so to change the state of the rings, edit the relevant YAML file (found in
modules/swift/files/CLUSTERNAME_hosts.yaml), and put through a CR as usual.
hosts.yaml file must define one or more schemes, which specify which devices belong to which rings, and with what weight. You typically won't have to edit these, as WMF tends to buy hardware of consistent specification. Each scheme is a mapping of ring name to the members of that ring, with an additional member "weight" which contains a mapping of ring name to the weight of members of that ring. For example:
schemes: prod: objects: [sdc1, sdd1, sde1] accounts: &ap [sda3, sdb3] containers: *ap ssds: &sp [sda4, sdb4] weight: objects: 4000 accounts: &acw 100 containers: *acw ssds: 300
Here, there is a single scheme "prod". There are three devices (
/dev/sde1) which hold objects, with weight 4000. In common with the previous Swift ring management code, the rings are given friendly names ("objects" for the "object" ring, "ssds" for the "object-1" ring, etc.); the mapping of friendly name to ring is defined in the
Adding a host
If a host is the same type as other hosts in the cluster, simply add the host to the relevant list in hosts.yaml. So
hosts: prod: - hostname
hosts: prod: - hostname - newhostname
newhostname, a new host with the
prod device scheme. Remember to also add the new node to
Removing a host
Simply removing the host entry from hosts.yaml will cause it to be immediately removed from the rings; it is generally better to drain it first (i.e. gradually remove weight from all of its devices). This can be done thus:
hosts: prod: - hostname: - drain
failed instead of
drain will cause all weights to be set to 0 immediately, rather than gradually.
Removing a device
Do this by setting its weight to zero. By default, this will take effect gradually; if you want to do so straight away, specify
immediate as well as weight zero. As a convenience, you can also use
drain for "weight 0" and
failed for "weight 0, immediate". For example:
hosts: prod: - hostname: - sdc1: 0 - sdd1: [0, immediate] - sde1: failed
/dev/sdc1 will have weight on it gradually reduced to 0, whereas
/dev/sde1 will both have their weight set to 0 immediately.
Adjusting a device weight
This is essentially the same as the previous operation, but you can specify any weight you like. You may also specify
immediate for a change you want to not be made gradually. For example:
hosts: prod: - hostname: - sdc1: 2000 - sdd1: [3500, immediate]
/dev/sdc1 will have its weight gradually changed to 2000, and
/dev/sdd1 will have its weight immediately set to 3500.
Typically, you don't need to do this - it should be run automatically on the cluster's ring manager node, which will make changes and distribute the ring files for you. You can run it yourself to see what changes it would make, however - the default operation is to merely say what changes if any would be made. The help message details all the command-line arguments:
usage: swift_ring_manager.py [-h] [--doit] [--immediate-only] [--update-tmp-rings] [-v] [-c CONF_FILE] [--ring-dir RING_DIR] [-o OUT_DIR] [--skip-dispersion-check] [--skip-replication-check] [-s] [--host-file-path HOST_FILE_PATH] [--generate-host-file] Swift Ring Manager optional arguments: -h, --help show this help message and exit --doit Update ring files and deploy --immediate-only Only make changes flagged as immediate --update-tmp-rings Only update temporary copies of ring files -v, --verbose emit more debugging information -c CONF_FILE, --conf-file CONF_FILE path to configuration file --ring-dir RING_DIR directory containing ring builder files -o OUT_DIR, --out-dir OUT_DIR directory to put new rings into --skip-dispersion-check don't check dispersion OK before making changes --skip-replication-check don't check replication OK before making changes -s, --syslog Log to syslog rather than stdout --host-file-path HOST_FILE_PATH fake hosts list file path --generate-host-file generate a hosts file for future testing use