You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Backup-mariadb

From Wikitech-static
Revision as of 08:09, 23 September 2020 by imported>Jcrespo (+cat)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

backup-mariadb (previously backup_mariadb.py) is a Python 3 script that allows generating MariaDB backups of different formats (right now mydumper and xtrabackup (mariabackup) are implemented, but can be easily extended with others.

Note that while backup-mariadb can call xtrabackup, it can only do so for a local mariadb instance, so normally it is used in combination with transfer.py, which uses cumin to transfer backups from a remote host, and later postprocess it with backup-mariadb.

backup-mariadb can be used with options on command line, but it is intended to be used more easily with a configuration file.

Usage

backup_mariadb.py is installed on PATH on dbprov* hosts (via Puppet), and it is used to generate weekly dumps, as well as postprocess remote snapshots (xtrabackup) generated by transfer.py.

dbprov1001$ backup-mariadb -h
usage: backup_mariadb.py [-h] [--config-file CONFIG_FILE] [--host HOST]
                         [--port PORT] [--user USER] [--password PASSWORD]
                         [--threads THREADS] [--type {dump,snapshot}]
                         [--only-postprocess] [--rotate]
                         [--retention RETENTION] [--backup-dir BACKUP_DIR]
                         [--rows ROWS] [--archive] [--compress]
                         [--regex REGEX] [--stats-host STATS_HOST]
                         [--stats-port STATS_PORT] [--stats-user STATS_USER]
                         [--stats-password STATS_PASSWORD]
                         [--stats-database STATS_DATABASE]
                         [section]

Create a mysql/mariadb logical backup using mydumper or a snapshot using
mariabackup.It has 2 modes: Interactive, where options are received from the
command line and non-interactive, where it reads from a config file and
performs several backups

positional arguments:
  section               Section name of the backup. E.g.: "s3", "tendril". If
                        section is set, --config-file is ignored. If it is
                        empty, only config-file options will be used and other
                        command line options will be ignored. For --only-
                        postprocess work, you can also provide an absolute
                        path of the directory

optional arguments:
  -h, --help            show this help message and exit
  --config-file CONFIG_FILE
                        Config file to use. By default, /etc/wmfbackups/backups.cnf
                        .
  --host HOST           Host to generate the backup from. Default: localhost.
  --port PORT           Port to connect to. Default: 3306.
  --user USER           User to connect for backup. Default: root.
  --password PASSWORD   Password used to connect. Default: empty password.
  --threads THREADS     Number of threads to use for exporting. Default: 18
                        concurrent threads.
  --type {dump,snapshot}
                        Backup type: dump or snapshot. Default: dump
  --only-postprocess    If present, only postprocess and perform the metadata
                        gathering metrics for the given ongoing section
                        backup, skipping the actual backup. Default: Do the
                        whole process.
  --rotate              If present, run the rotation process, by moving it to
                        the standard."latest" backup. Default: Do not rotate.
  --retention RETENTION
                        If rotate is set, purge backups of this section older
                        than the given value, in days. Default: 18 days.
  --backup-dir BACKUP_DIR
                        Directory where the backup will be stored. Default:
                        /srv/backups.
  --rows ROWS           Max number of rows to dump per file. Default: 20000000
  --archive             If present, archive each db on its own tar file.
  --compress            If present, compress everything into a tar.gz.Default:
                        Do not compress.
  --regex REGEX         Only backup tables matching this regular
                        expression,with format: database.table. Default: all
                        tables
  --stats-host STATS_HOST
                        Host where the statistics database is.
  --stats-port STATS_PORT
                        Port where the statistics database is. Default: 3306
  --stats-user STATS_USER
                        User for the statistics database.
  --stats-password STATS_PASSWORD
                        Password used for the statistics database.
  --stats-database STATS_DATABASE
                        MySQL schema that contains the statistics database.

See also