You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Orchestrator
- You may also be looking for the WikiFunctions function-orchestrator.
Orchestrator is a service for managing mysql cluster replication. The data-persistence SRE team is currently doing a proof-of-concept deployment of it within WMF, with the aim of replacing Tendril/Dbtree.
Operations
Adding a section to orchestrator
- Deploy the orchestrator grants to the section (
modules/role/templates/mariadb/grants/orchestrator.sql.erb
in the puppet repo). This should be done on the active DC's primary instance, and also on both DC's sanitarium hosts. - Clean up the heartbeat table so that there's no stale entries.
- E.g. run this against all instances individually:
set session sql_log_bin=0; delete from heartbeat where server_id=171974662 limit 1
- E.g. run this against all instances individually:
- Add the primary instance to orchestrator.
- Ssh to the dborch node, and run
sudo orchestrator -c discover -i FQDN
- N.B. it needs to be the FQDN of the instance.
- Ssh to the dborch node, and run
Upgrading orchestrator
Orchestrator automatically deploys schema changes when it gets upgraded. It tracks these in the orchestrator_db_deployments
table. On startup it will check to see if the current version number is in that table, and if not it will perform all schema changes. It will not detect if a later version has been deployed. This means that we need a full backup of the orchestrator database before doing an upgrade, as otherwise we do not have a way to rollback.
Packaging
Updating orchestrator packaging to a new upstream version
- Check out the orchestrator package repo: https://gerrit.wikimedia.org/r/admin/repos/operations/debs/orchestrator
- On the
master
branch, run./debian/repack v$VER
. Note the leadingv
in the upstream version umber. This will create a tarball in the current directory. - Move the tarball out of the git working dir:
mv orchestrator_$VER.orig.tar.xz ..
- Import it:
gbp import-orig ../orchestrator_$VER.orig.tar.xz
. This will add a commit to theupstream
branch a newupstream/$VER
tag referencing it. It will then merge the newupstream
branch intomaster
. - Push these new branches directly to gerrit, as they are not reviewable:
git checkout upstream; git push; git push origin upstream/$VER
git checkout master; git push
- Create a debian changelog entry for the new version:
dch -D buster-wikimedia --force-distribution -v $VER-1
. If you forget to do this, trying to build a package will fail horribly withdpkg-source: error: unrepresentable changes to source
- Test building the package to make sure that still works, and then send a CR for review with your changes.
Creating a new orchestrator release
You will need a gpg key to sign the new release. git tag
will prompt you for your gpg password when creating the new tag.
- For simplicity, set 2 environment variables in your shell,
$DEBVER
for the new release you're creating, and$OLDDEBVER
for the previous release. E.g.:DEBVER=3.2.6-1; OLDVER=3.2.3-3
- Add/update a debian changelog entry for
$DEBVER
. Send a CR for review for any changes.- If it doesn't already exist, create it with
dch -D buster-wikimedia --force-distribution -v ${DEBVER:?}
- If it doesn't already exist, create it with
- Create a git tag for the release, and populate it with changes made since the last release:
git tag -s -a -F <(echo orchestrator ${DEBVER:?}; echo; git log --no-decorate --oneline debian/${OLDDEBVER:?}..) -e debian/${DEBVER:?}
. This will prompt you for a gpg password to sign the tag with. - Check that the new tag looks good:
git show debian/${DEBVER:?}
- Push the tag to the upstream repo:
git push origin debian/${DEBVER:?}
Building orchestrator packages
It's not currently possible to build orchestrator on deneb due to its golang version requirements. (Until Puppet host certs do not contain Subject Alt Name entries is fixed, or a workaround implemented, we're limited to golang 1.14).
- Check out the orchestrator package repo: https://gerrit.wikimedia.org/r/admin/repos/operations/debs/orchestrator
- Install the following prerequisites:
sudo apt install devscripts debhelper dh-golang
- golang 1.14
- Build with
debclean -d && debuild -d -us -uc
(-d
is needed to work around the fact that the build requirement on golang 1.14 isn't being satisfied by a debian package).
Uploading new orchestrator packages
This is a simplified version of Debian Packaging#Upload to Wikimedia Repo.
- On
apt1001
:mkdir -p ~/orchestrator && rm ~/orchestrator/*.changes
- From your build dir on your local machine:
scp ../*changes ../*deb ../*dsc apt1001.eqiad.wmnet:orchestrator/
- Back on
apt1001
:cd orchestrator && sudo -i reprepro -C main include buster-wikimedia *.changes
- In
#wikimedia-operations
on irc:!log uploaded orchestrator $VERSION packages to apt.wm.o (buster) TXXXXXX
Troubleshooting
Entry in database_resolve that maps to a bare hostname
+--------------------+--------------------+---------------------+ | hostname | resolved_hostname | resolved_timestamp | +--------------------+--------------------+---------------------+ | pc1008.eqiad.wmnet | pc1008 | 2020-11-18 10:11:58 | +--------------------+--------------------+---------------------+
This can cause a 'ghost' cluster to appear, containing the bare-hostname version of the host. To fix this:
systemctl stop orchestrator orchestrator -c forget -i <instance> for all instances in the ghost cluster orchestrator -c reset-hostname-resolve-cache systemctl start orchestrator
Stopping orchestrator is required to stop it from reinserting the bad entry into hostname_resolve.
The entries can be queried via orchestrator -c show-resolve-hosts