You are browsing a read-only backup copy of Wikitech. The live site can be found at

MariaDB/Decommissioning a DB Host

From Wikitech-static
< MariaDB
Revision as of 15:08, 6 November 2020 by imported>LSobanski (Created page with "== Prerequisites: == * SSH access to cumin1001.eqiad.wmnet to depool + run the decommissioning script * SSH access to puppetmaster1001.eqiad.wmnet to merge puppet changes * [O...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


Decommissioning workflow:

Create a tracking ticket

  1. Create a decommission ticket with the following template example:
  2. If there is hardware problems, please specify so for the DCOps to label it so we do not re-use broken pieces.

Depool the host

  1. SSH to cumin1001
  2. dbctl instance HOSTNAME depool && dbctl config commit -m "Depool db1091 TASKNUMBER"

Remove the host from dbctl

  1. Create a puppet patch (example:
  2. SSH to puppetmaster1001
  3. sudo -i
  4. puppet-merge - if you see any changes other than yours here, contact the owners to see if these are ok to merge
  5. SSH to cumin1001
  6. dbctl config commit -m "Remove HOSTNAME from dbctl TASKNUMBER"

Remove all other puppet entries

  1. Create a puppet patch (example:

[Only codfw hosts, until they are migrated to Netbox] Remove DNS production entries

  1. Create a puppet patch (example:

Run the decommissioning script

  1. SSH to cumin1001
  2. sudo -i
  3. cookbook sre.hosts.decommission -t TASKNUMBER HOSTNAME.DC.wmnet
  4. Enter console password from Pwstore

Merge puppet + DNS changes

  1. SSH to puppetmaster1001
  2. sudo -i
  3. puppet-merge - if you see any changes other than yours here, contact the owners to see if these are ok to merge

Remove host from tendril and zarcillo

  1. Log the action in IRC (#wikimedia-operations) - !log Removing HOSTNAME from tendril and zarcillo TASKNUMBER
  2. SSH to cumin1001
  3. sudo -i
  4. Tendril:
    1. cd /home/marostegui/git/tendril/bin/
    2. for i in HOSTNAME(s); do echo $i; ./ $i.DC.wmnet 3306 ~/.my.cnf.tendril tendril | mysql -h db1115.eqiad.wmnet tendril; done - this can take a while sometimes, depending on how busy tendril is
  5. Zarcillo
    1. -h db1115 -A zarcillo
    2. Execute the following queries in the MySQL prompt (remember about the semicolon):
      1. delete from servers where hostname="HOSTNAME";
      2. delete from instances where name='HOSTNAME';
      3. delete from section_instances where instance like 'HOSTNAME';

Update the task and send it to dcops

  1. mark all the steps for "step for service owners" on:
  2. Reassign:
    • for eqiad to wiki_willy
    • for codfw to papaul
  3. Remove #DBA tag and add #dc-ops and #ops-eqiad OR #ops-codfw.
  4. Add the following comment: "This host is ready for DC-Ops to decommission".