You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
The beta cluster currently (May 2021) has a pair of MariaDB instances in a fairly standard master-replica setup running Buster and MariaDB 10.4.
Creating new instances
- Create a new
g3.cores8.ram16.disk20instance in horizon named deployment-dbNN where NN is an unused number, add a large enough Cinder volume and mount it to /srv, get its puppet certs signed, etc.
- TODO: determine if the VM needs that many resources. We used cores8.ram16 on g2 since we needed the extra disk, Cinder is now a thing.
On New Replica
mariadb::config::basedir: /opt/wmf-mariadb104in hiera, adjust the version number based on the used MariaDB version
- ensure puppet runs cleanly
/opt/wmf-mariadb104/scripts/mysql_install_db --user=mysql --basedir=/opt/wmf-mariadb104 --datadir=/srv/sqldata
nc -l -p 9210 | mbstream -x
On Existing DB
mariabackup --innobackupex --stream=xbstream /srv/sqldata --user=root --host=127.0.0.1 --slave-info | nc NEW-SERVER 9210
On New replica, After mariabackup
Do this before starting mysql!
mariabackup --innobackupex --apply-log --use-memory=10G /srv/sqldata
chown -R mysql: /srv
systemctl start mariadb
- tail the error log in /srv/sqldata and check for any errors
- If this is newer version of MariaDB, run
/opt/wmf-mariadb104/bin/mysql_upgrade --host=127.0.0.1to ensure compatibility.
- Check /srv/sqldata/xtrabackup_slave_info on the new host, it provides a partial CHANGE MASTER statement based on whichever host the target was replicating from. If this is correct, use this, adding MASTER_HOST and the repl account/password portions that are missing.
CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='...', MASTER_PORT=3306, MASTER_HOST='deployment-db1', MASTER_LOG_FILE='deployment-db1-bin.000026', MASTER_LOG_POS=191406295;
- The master repl password should be in the file /var/lib/git/labs/private/modules/secret/secrets/mysql/repl_password on the deployment-puppetmaster (currently
- If you ran the hot backup from the master, use the information in /srv/sqldata/xtrabackup_binlog_info as slave_info won't exists.
- If you copied from the secondary master, xtrabackup_slave_info will point to the active master; use xtrabackup_binlog_info instead.