You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

MariaDB/Upgrading a section: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>LSobanski
mNo edit summary
imported>Ladsgroup
 
(5 intermediate revisions by 3 users not shown)
Line 18: Line 18:
* Run puppet on install1003 and install2003
* Run puppet on install1003 and install2003
* Depool the host (if needed) using software/dbtools/depool-and-wait
* Depool the host (if needed) using software/dbtools/depool-and-wait
* Silence the host in Icinga
* Silence the host in Icinga (e.g. on a cumin host, <code>cookbook sre.hosts.downtime xxxx.wmnet -D1 -t TXXXXXX -r "reimage for upgrade - TXXXXXX"</code>)
* Stop MySQL on the host
* Stop MySQL on the host
* Run <code>umount /srv; swapoff -a</code>
* Run <code>umount /srv; swapoff -a</code>
* Run reimage: <code>sudo -E wmf-auto-reimage xxxx.wmnet -p TXXXXXX</code>
* Run reimage: <code>sudo -E sudo cookbook sre.hosts.reimage xxxx.wmnet -p TXXXXXX</code>
* Wait until the host is up  
* Wait until the host is up  
* Run <code>systemctl set-environment MYSQLD_OPTS=”--skip-slave-start”</code>
* Run <code>systemctl set-environment MYSQLD_OPTS=”--skip-slave-start”</code>
* Run <code>systemctl start mariadb ; mysql_upgrade</code>
* Run <code>chown -R mysql. /srv/*; systemctl start mariadb ; mysql_upgrade</code>
* Run <code>systemctl restart prometheus-mysqld-exporter.service</code>
* Run <code>systemctl restart prometheus-mysqld-exporter.service</code>
* Dropped the host from Tendril and re-add it, otherwise they won’t get updated on tendril metrics
* Dropped the host from Tendril and re-add it, otherwise they won’t get updated on tendril metrics
Line 33: Line 33:
* Wait until the host is up
* Wait until the host is up
* Repool the host.
* Repool the host.
== Upgrading mariadb minor version ==
The general steps for restarting a mysql instance apply, except there is no need to reboot the host entirely, unless a kernel upgrade requires it. Follow the details at: [[MariaDB/Rebooting_a_host]], '''including how to safely depool,  shutdown all servers and disable monitoring alerting'''.
The main changes are:
# You should log that a maintenance is about to happen:<syntaxhighlight lang="irc">
!log Upgrade db1111 T123456
</syntaxhighlight>
# The package for the mariadb server must be upgraded, usually:<syntaxhighlight lang="shell">
sudo apt upgrade 'wmf-mariadb*'
</syntaxhighlight>where wmf-mariadb* is the package version you want to upgrade to, e.g. wmf-mariadb104, for WMF's version of MariaDB 10.4 WMFf package is built thinking to avoid side effects- so '''it won't automatically try to stop, restart or alter in any way a running instance'''- so it is possible to run it an any time, even if a previous version is currently executing. But unless there is a reason for it (e.g. minimizing upgrade downtime) it should probably ran after all current instances are shutdown.
# Start mysql in a safe way- not starting replication automatically and removing any old buffer pool dump:<syntaxhighlight lang="shell">
sudo systemctl set-environment MYSQLD_OPTS="--skip-slave-start"
<for each datadir> sudo mv ib_buffer_pool ib_buffer_pool.bak
</syntaxhighlight>
# mysql_upgrade must be ran on every instance after startup, and before replication starts, for single instance hosts:<syntaxhighlight lang="shell">
systemctl start mariadb
systemctl status mariadb  # check it started correctly (it is ok to have some errors on first start up due to ongoing upgrade, due to old table formats)
mysql_upgrade
</syntaxhighlight>For multiple instance hosts, '''for each instance''':<syntaxhighlight lang="shell">
sudo systemctl start mariadb@<section>
sudo systemctl status mariadb@<section>  # check it started correctly (it is ok to have some errors on first start up due to ongoing upgrade, due to old table formats)
sudo mysql_upgrade -S /run/mysqld/mysqld.<section>.sock
</syntaxhighlight>Where section is the list of instances to upgrade on that host (e.g. s1 and s2, x1, s5 and s4, etc.)
# After upgrade, if the mysql database changed, it is important to perform a reboot. This is normally skippable for minor upgrades, but guarantees it started with the right formatting: <syntaxhighlight lang="shell">
sudo systemctl restart mariadb # or sudo systemctl restart mariadb@<section> (for each section upgraded)
</syntaxhighlight>
The rest of steps to get the server into production state would be the same as on a regular reboot/restart: [[MariaDB/Rebooting_a_host]] (restart replication, repool, reenable monitoring, safety checks)


{{SRE/Data Persistence/Footer}}
{{SRE/Data Persistence/Footer}}
[[Category:MariaDB]]

Latest revision as of 14:06, 25 January 2022

Order of upgrades

  • Upgrade clouddb* hosts.
  • Upgrade Sanitarium hosts in both DCs
  • Upgrade Sanitarium primaries in both DCs and ensure sanitarium host hangs from the 10.4 one in the active DC
  • Upgrade the candidate master on the standby DC
  • Upgrade the backup source in the standby DC (coordinate with Jaime)
  • Upgrade the master in the standby DC
  • Upgrade the candidate master in the primary DC
  • Upgrade the backup source in the primary DC (coordinate with Jaime)
  • Switchover the primary host in the primary DC to a Buster+10.4 host
  • Upgrade the old primary and make it a candidate primary

Upgrade procedure

  • Patch the dhcp file: [example]
  • Run puppet on install1003 and install2003
  • Depool the host (if needed) using software/dbtools/depool-and-wait
  • Silence the host in Icinga (e.g. on a cumin host, cookbook sre.hosts.downtime xxxx.wmnet -D1 -t TXXXXXX -r "reimage for upgrade - TXXXXXX")
  • Stop MySQL on the host
  • Run umount /srv; swapoff -a
  • Run reimage: sudo -E sudo cookbook sre.hosts.reimage xxxx.wmnet -p TXXXXXX
  • Wait until the host is up
  • Run systemctl set-environment MYSQLD_OPTS=”--skip-slave-start”
  • Run chown -R mysql. /srv/*; systemctl start mariadb ; mysql_upgrade
  • Run systemctl restart prometheus-mysqld-exporter.service
  • Dropped the host from Tendril and re-add it, otherwise they won’t get updated on tendril metrics
  • Check all the tables before starting replication (this can take up to 24h depending on the section)
    • In a screen run: mysqlcheck --all-databases
    • If any corruption is discovered, fix it with the following: journalctl -xe -u mariadb | grep table | grep Flagged | awk -F "table" '{print $2}' | awk -F " " '{print $1}' | tr -d "\`" | uniq >> /root/to_fix ; for i in `cat /root/to_fix`; do echo $i; mysql -e "set session sql_log_bin=0; alter table $i engine=InnoDB, force"; done
  • Start the replica
  • Wait until the host is up
  • Repool the host.

Upgrading mariadb minor version

The general steps for restarting a mysql instance apply, except there is no need to reboot the host entirely, unless a kernel upgrade requires it. Follow the details at: MariaDB/Rebooting_a_host, including how to safely depool, shutdown all servers and disable monitoring alerting.

The main changes are:

  1. You should log that a maintenance is about to happen:
    !log Upgrade db1111 T123456
    
  2. The package for the mariadb server must be upgraded, usually:
    sudo apt upgrade 'wmf-mariadb*'
    
    where wmf-mariadb* is the package version you want to upgrade to, e.g. wmf-mariadb104, for WMF's version of MariaDB 10.4 WMFf package is built thinking to avoid side effects- so it won't automatically try to stop, restart or alter in any way a running instance- so it is possible to run it an any time, even if a previous version is currently executing. But unless there is a reason for it (e.g. minimizing upgrade downtime) it should probably ran after all current instances are shutdown.
  3. Start mysql in a safe way- not starting replication automatically and removing any old buffer pool dump:
    sudo systemctl set-environment MYSQLD_OPTS="--skip-slave-start"
    <for each datadir> sudo mv ib_buffer_pool ib_buffer_pool.bak
    
  4. mysql_upgrade must be ran on every instance after startup, and before replication starts, for single instance hosts:
    systemctl start mariadb
    systemctl status mariadb  # check it started correctly (it is ok to have some errors on first start up due to ongoing upgrade, due to old table formats)
    mysql_upgrade
    
    For multiple instance hosts, for each instance:
    sudo systemctl start mariadb@<section>
    sudo systemctl status mariadb@<section>  # check it started correctly (it is ok to have some errors on first start up due to ongoing upgrade, due to old table formats)
    sudo mysql_upgrade -S /run/mysqld/mysqld.<section>.sock
    
    Where section is the list of instances to upgrade on that host (e.g. s1 and s2, x1, s5 and s4, etc.)
  5. After upgrade, if the mysql database changed, it is important to perform a reboot. This is normally skippable for minor upgrades, but guarantees it started with the right formatting:
    sudo systemctl restart mariadb # or sudo systemctl restart mariadb@<section> (for each section upgraded)
    

The rest of steps to get the server into production state would be the same as on a regular reboot/restart: MariaDB/Rebooting_a_host (restart replication, repool, reenable monitoring, safety checks)



This page is a part of the SRE Data Persistence technical documentation
(go here for a list of all our pages)