Transfer.py

transfer.py is a Python 3 script intended to move large files or directory trees between WMF production hosts in an efficient way, initially thought for database maintenance, but which can be used to move arbitrary files.

Context

Before there was a backup and provisioning workflow, new database hosts (or hosts that needed rebuild, e.g. after a crash) were setup in different, completely manual ways. Mostly using netcat.

Before cumin was available, work started on a way to automate that (task T156462) with the aim to add a programmable api, consistency, speedup, handling the firewall, and generally stop doing things manually. Used at first to perform cold database copies.

Once cumin was deployed, and logical backups were in place, transfer.py was extended to be used for hot database backups and recoveries, as a building block of the recovery system.

Technical details

The original spec for a transfer script was using multicast or bittorrent protocols for fast recovery of multiple hosts. For the time, however, basically cumin is run so that netcat is run on source, and another instance listening on target host, and either a single file or a tar copy of a directory is piped through. Optionally, compression with pigz can be used, as well as encryption with openssl. Pigz can improve enormously the speed of the transfer, as database can be compressed as much as 5 times, reducing the total bandwidth used. Data can also be checksummed, but that adds some overhead at the beginning of the transfer.

At the Wikimedia Foundation infrastructure, cumin is being used as the remote execution framework, but others are also available and can be made to work. However, for things like mysql transfers, certain things like mysql port assignation and paths are assumed to be in a specific places.

For MySQL, two modes (in addition to the default "file") were added:

• xtrabackup, where the source of the backup is not obtained from the filesystem, but from a mariabackup run with streaming (xbstream) from a running MariaDB server. In this mode, the mysql server socket is used as the origin path. It can optionally stop replication in cases where that may speed up the copy/preparation process. It only transfer the files, it does not prepare or touch in any way the generated files (that is considered out of scope of this script, as one may want to wait to do that for incremental backups or other reasons).
• decompress, where the source is a precompressed tar.gz file containing a single directory, intended with the same format as the snapshot tarballs generated by the backup system (prepared xtrabackup datadirs), and result in a decompressed directory (very similar to "file" operation, but without having to compress on source)

In addition to the automation of all commands (iptables, tar, openssl, pigz, nc, remote execution), transfer.py does important sanity checks, like making sure it does not overwrite existing files (it aborts before transfer) and making sure it has enough disk space on the selected directory.

Dependencies

transfer.py requires the following technologies

• Python 3, preferable 3.5 or later
  • cumin python class if chosen as the transfer system
• A remote execution system (ssh, paramiko, salt, etc.). If none are available, there is a LocalExecution class, but it will only allow to run commands locally (local transfers)
  • For cumin, transfer.py must be installed on a cumin* host to be able to execute remote commands
• Netcat (nc)
• pigz for compression
• tar for archiving before streaming
• openssl for encryption
• du, df to calculate used and available disk size
• bash to pipe the different unix commands
• wmf-mariadb package and an instance running for --type=xtrabackup
• xtrabackup (mariabackup) installed locally on the mariadb hosts for --type=xtrabackup
• mysql client if replication wants to be stopped
• iptables to manage the firewall hole during transfer

Note: transfer.py expect the user to have root privileges without the sudo prefix.

Usage

transfer.py is installed (via Puppet) on PATH on WMF production infrastrcture on cumin1001 and cumin1002, and has to run as root (like cumin).

cumin1001$ transfer.py -h
usage: transfer.py [-h] [--port PORT] [--type {file,xtrabackup,decompress}]
                   [--compress | --no-compress] [--encrypt | --no-encrypt]
                   [--checksum | --no-checksum] [--stop-slave]
                   source target [target ...]

positional arguments:
  source - fully qualified domain of the host where the files to be copied are currently located, the symbol ':', and a file
           or directory path of such files (e.g. sourcehost.wm.org:/srv ). There can be only one source host and path.
  target - fully qualified domain of the hosts (separated by spaces) where the files will be copied, each one with its 
           destination absolute path directory, separated by ':'. There must be at least one target. If more than one target 
           is defined, it will be copied to all of them (at the moment, in a serial way).
           The target path MUST be a directory. No matter if it is a file or a directory, the copied source path will be 
           copied inside it. If the source is a file, the file will be transmitted as is. If the source is a directory, 
           it and all its contents will be copied to the target. This behavior is similar and inspired by scp.
           For example:
           transfer.py sourcehost.wikimedia.org:/etc/my.cnf otherhost.wikimedia.org:/tmp otherhost2.wikimedia.org:/srv/tmp
           will result on /tmp/my.cnf on outherhost and /srv/tmp/my.cnf
           transfer.py sourcehost.wikimedia.org:/etc otherhost.wikimedia.org:/tmp otherhost2.wikimedia.org:/srv/tmp
           will result on all the /etc files and subdirectories ending up on /tmp/etc and /srv/tmp/etc respectively.
           In case of an xtrabackup copy, the path must point to the socket of the instance to prevent ambiguity of
           which instance to copy if the source has multiple mysql instances running. For example:
           transfer.py --type=xtrabackup db1051.eqiad.wmnet:/run/mysqld.sock dbprov1005.eqiad.wmnet:/srv/backups/snapshots/ongoing
           In the case of decompress copy, the source patch must be the .tar.gz file, and the target the datadir (it assumes the
           tar.gz. will have an arbitrary named directory, and it extrats its contents, it will not create a datadir).

optional arguments:
  -h, --help            show this help message and exit
  --port PORT           Port used for netcat listening on the source. By default, 4444, but it must be changed if more 
                        than 1 transfer to the same host happen at the same time, or the second copy will fail top open
                        the socket again. This port has its firewall disabled during transfer automatically with an extra
                        iptables rule.
  --type {file,xtrabackup,decompress}
                        File: regular file or directory recursive copy
                        xtrabackup: runs mariabackup on source
                        decompress: a tarball is transmitted as is and decompressed on target
  --compress            Use pigz to compress stream using gzip format (ignored on decompress mode)
  --no-compress         Do not use compression on streaming
  --encrypt             Enable compression using openssl and algorithm chacha20 (default)
  --no-encrypt          Disable compression- send data using an unencrypted stream
  --checksum            Generate a checksum of files before transmission which will be used for checking integrity after
                        transfer finishes. It only works for file transfers, as there is no good way to checksum a running
                        mysql instance or a tar.gz
  --no-checksum         Disable checksums
  --stop-slave          Only relevant if on xtrabackup mode: attempt to stop slave on the mysql instance before running 
                        xtrabackup, and start slave after it completes to try to speed up backup by preventing many changes
                        queued on the xtrabackup_log. By default, it doesn't try to stop replication.
  --verbose             Relevant information about transfer + information about Cuminexecution is shown as output.
                        By default, the output contains only relevant information about the transfer.

Obtaining

transfer.py is a script in the transferpy subdirectory of the wmfmariadbpy repo Gerrit.

What's New? (GSoC-2020)

Now the transfer framework moved to its own module named transferpy. This framework has 3 basic modules and a RemoteExecution:

• Transferrer: The Transferrer class is responsible for the act on the user arguments and make the send/receive possible.
• Firewall: The Firewall class is for open/close the ports in the iptables in order to receive the data by the receiver machines.
• MariaDB:
• RemoteExecution: The RemoteExecution is the module responsible for the execution of command on the remote machines. transfer framework mainly uses the Cumin execution.

Wishlist and know issues

• The encryption impacts very negatively in performance, and it is not forward-secret. A low penalty alternative with forward secrecy should be used instead
• Sizes are calculated with du, which is known to produce different results on different host even if the copy has been accurate. This is why the size check gives only a warning if it shows a difference on source and target hosts. A different, more reliable method could be used, but may take more resources.
• Checksum happens in a previous step before transfer- it would be nice to run checksumming in parallel (or at the same time) with transfer so it doesn't impact its latency and it is not normally disabled
• Configurable compression by using other algorithms depending on the data (e.g. lz if compression speed is not the limiting factor, etc.)
• Multicast, torrent or other solution should be setup to allow parallel transmission of data to multiple hosts in an efficient manner
• Better logging and error checking
• In general, more flexibility (e.g. level of parallelism, etc.) as long as it uses by default or autodetects saner defaults to not increase too much the difficulty of usage
• Firewall whole opening should be optional
• It should check that a port is available before binding to it
• It should also wait until port is fully opened (polling), instead of just waiting 3 seconds

See also

• Cumin
• More examples of using transfer.py for generating and recovering mysql backups