Portal:Data Services/Admin/Runbooks/Depool wikireplicas
Wikireplicas need work sometimes, fail, etc. This details how to depool each stage of the path to a wikireplica database.
In order to reach a wikireplica server from Wikimedia Cloud Services, you cross three proxy layers. That provides several opportunities for depooling and reshuffling parts of the system for maintenance and down time. Operations take place at the proxy layers or, potentially, in DNS.
The VM proxies hold 8 IPs each so that they can route requests coming into port 3306 (mysql) according to DNS. The mysql wire protocol does not allow you to identify a hostname or database before authenticating to the server, partly because the server talks first. You have to hit the right server for handshake. So that users can select their database using the set of DNS CNAMES back to one of the 8 section (mariadb instance) names of s1 through s8, you need 8 IPs on a proxy. This layer of haproxy interprets an IP into a port on the next layer (since each section corresponds to a non-standard port number).
At this time, you can depool a VM proxy quite simply without changing anything in OpenStack by editing modules/openstack/files/util/wikireplica_dns.yaml in puppet so that analytics and web IPs for the various sections all match the VM proxy you are not depooling.
Next, as root on a cloudcontrol host:
puppet agent -tto get your puppet change
This would direct all traffic at the new proxy (just as soon as DNS caches expire and all existing connections are done). This could take a fair while. If you use this method, waiting an hour is not a bad idea once the change is in place and you have run wmcs-wikireplica-dns. After you've waited a while, you can be sure that connections will disconnect if you stop haproxy on the depooled node.
Repool by reverting your puppet patch and doing the same steps above!
WMCS shouldn't be concerned with the LVS layer since it is a straight passthrough. We would normally depool a node upstream of that by just using the method in the next section, not using pybal. The only concern here is that someone may be alerted for those when a dbproxy server goes down, but I don't *think* they are monitored. While operations don't take place here, this layer is important to figuring out the next bit because it attributes public IPs to the loopback interface of each dbproxy server. See below.
These servers (currently dbproxy1018.eqiad.wmnet and dbproxy1019.eqiad.wmnet) are the last layer before you actually hit a server. They have LVS public IPs as well as private IPs attributed to them. If you ssh to one and run
ip a show dev lo you'll see the ip for that server. You'll also see that the IP is repeated 8 times under
profile::wmcs::db::wikireplicas::section_backends hiera key in prefix puppet on one of clouddb-wikireplicas-proxy-1 or clouddb-wikireplicas-proxy-2 "prefixes". The reason is that every section is going to the appropriate port on one proxy or the other in normal operation. To drop one of them from operation, simply find the VM proxy hiera that has the public IP for the dbproxy hardware proxy that you want to depool, and swap out that IP for the one that is not being depooled.
When this is saved out, haproxy will reload on puppet run. You can run puppet yourself just to be sure on the clouddb-wikireplicas-proxy-? server that you changed hiera for. This will direct all new connections to the other hardware proxy. This will not cut off existing connections unless you restart haproxy, which is likely fine for most cases, since connections should not be longer than an hour or so even on the analytics proxy.
Wikireplica database servers
Finally Wikireplica database servers are generally depooled using puppet host hiera that is applied in operations/puppet to the dbproxy hardware server in question. Remember that most database servers include 2 mariadb instances (sections) for each server. The configuration of the hardware proxy layer is what allows depooling of database servers (and other manipulation of traffic).
The basic, default configuration is pulled from puppetdb based on puppetized settings of the database servers themselves. Each analytics database instance is the standby for the web service of the same section number in the default configs, and each web database instance is the standby for the analytics instance of the same section number, using some reduce and merge trickery in puppet. Depooling is accomplished by overriding what the reduce functions produce for the sections in the resulting hash using the
profile::mariadb::proxy::multiinstance_replicas::section_overrides key in host hiera, which will be merged at the end of the process.
Instructions and examples are provided in the comments for the host hiera yaml. For instance, see hieradata/hosts/dbproxy1018.yaml or hieradata/hosts/dbproxy1019.yaml in operations puppet. The big difference here is that you need to manually reload haproxy yourself once puppet runs.
At any time, you can see the effective, in-memory configuration and status of the haproxy server by running the following as root:
root@dbproxy1018:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio
Coordinate with the Data Persistence team to make sure there are no on-going operations you could interfere with, of course.
An example of doing this is provided in the comments of each proxy server's host hiera file to make it clearer and for quicker depooling. Once the appropriate hiera file is changed, you will need to do the same reloading process as for the legacy replicas for this to take effect.
As a basic example, if you wanted to depool clouddb1015.eqiad.wmnet specifically and it is ok to leave it as the secondary on dbproxy1018 (which it normally would be), then you'd add (or decomment) the following to hieradata/hosts/dbproxy1019.yaml and you'd need to reload haproxy on the given proxy, and this should do it by simply overwriting the s4 and s6 keys:
profile::mariadb::proxy::multiinstance_replicas::section_overrides: s4: clouddb1019.eqiad.wmnet: ipaddress: 10.64.48.9 s6: clouddb1019.eqiad.wmnet: ipaddress: 10.64.48.9
After you do merge your puppet patch, you log into the target dbproxy101x server and run:
sudo -i puppet agent -t
sudo systemctl reload haproxy
Again you can make sure everything did what you expected with the above socat command as root.
Sidenote on the overrides hiera
ipaddress, other valid keys are
weight, which only really matters if you have more than one host under the section key, the boolean
standby, which adds the host entry to haproxy as a standby under that key and
depooled, which expressly removes the host entry from the resulting file. The last one is unlikely to be necessary since adding an uncommented section key (eg. s2:) will overwrite the automatically generated one anyway. It could be useful one day, possibly if we start doing deep merges or something for the overrides. At this time, it is a shallow merge.
If you are following this, you are probably already a part of the WMCS or Data Persistence team. Perhaps you can ask the team you are not on if you need more help?