Jump to content

This is a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

MariaDB/Clone a host

From Wikitech

Derived from MariaDB/Provisioning a host

Good to know before starting

Please be sure to be aware of the following page(s):

Host preparation

Here is an overview of what a host depends on:

arrow hell

Move the host in the right groups with the right role

Only required if the host is new or it is changing its section
  1. In manifests/site.pp :
    1. Add the DBNAME to the appropriate regexps
    2. Remove the insetup role from DBNAME
  2. In hieradata/hosts/$destination_host.yaml add the configuration content of hieradata/hosts/$source_host.yaml . You can check this example if needed .

Add the host to dbctl config ( example )

Only required if the host is new

In conftool-data/dbconfig-instance/instances.yaml add - DBNAME in the appropriate location and then run dbctl commit command on cumin

Cloning

The sre.mysql.clone cookbook will:

  • Ensure the new host is configured in dbctl and puppet
  • Depool the source and target servers
  • Disable icinga monitoring for the target
  • Clone the source to target
  • Wait for replication and icinga monitoring to normalize
  • Pool-in the servers
  • Add the new host to zarcillo

See --help for more and try --dry-run on first use.

Clone via the sre.mysql.clone cookbook in a tmux/screen:

sudo cookbook sre.mysql.clone --source <source_fqdn> --target <destination_fqdn> --task <task_id>

Example: db1190 will be recloned from db1180 and will be configured to replicate from db1173.eqiad.wmnet

sudo cookbook sre.mysql.clone --source db1180.eqiad.wmnet --target db1190.eqiad.wmnet --task T1234

Monitor the run output

After a few days of uptime

If required (i.e. upon decomissionning) and if everything works properly, we can now depool our old host:

sudo dbctl instance $source_host depool

We will also repool the destination host with the right script which has to be executed in a tmux/screen from cumin :

./repool $destination_host 'Host warmup' 15 30 45 60 75 90 100

each value at the end of the command line is a "step" and a traffic percentage to repool with