You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Orchestrator
Jump to navigation
Jump to search
- You may also be looking for the WikiFunctions function-orchestrator.
Orchestrator is a service for managing mysql cluster replication. The data-persistence SRE team is currently doing a proof-of-concept deployment of it within WMF, with the aim of replacing Tendril/Dbtree.
Operations
Adding a section to orchestrator
- Deploy the orchestrator grants to the section (
modules/role/templates/mariadb/grants/orchestrator.sql.erb
in the puppet repo). This should be done on the active DC's primary instance, and also on both DC's sanitarium hosts. - Clean up the heartbeat table so that there's no stale entries.
- E.g. run this against all instances individually:
set session sql_log_bin=0; delete from heartbeat where server_id=171974662 limit 1
- E.g. run this against all instances individually:
- Add the primary instance to orchestrator.
- Ssh to the dborch node, and run
sudo orchestrator -c discover -i FQDN
- N.B. it needs to be the FQDN of the instance.
- Ssh to the dborch node, and run
Updating orchestrator packages to a new upstream version
- Check out the orchestrator package repo: https://gerrit.wikimedia.org/r/admin/repos/operations/debs/orchestrator
- On the
master
branch, run./debian/repack v$VER
. Note the leadingv
in the upstream version umber. This will create a tarball in the current directory. - Move the tarball out of the git working dir:
mv orchestrator_$VER.orig.tar.xz ..
- Import it:
gbp import-orig ../orchestrator_$VER.orig.tar.xz
. This will add a commit to theupstream
branch a newupstream/$VER
tag referencing it. It will then merge the newupstream
branch intomaster
. - Push these new branches directly to gerrit, as they are not reviwable:
git checkout upstream; git push; git push origin upstream/$VER
git checkout master; git push
- Create a debian changelog entry for the new version:
dch -D buster-wikimedia --force-distribution -v $VER-1
. If you forget to do this, trying to build a package will fail horribly withdpkg-source: error: unrepresentable changes to source
- Remember to change the distribution from
UNRELEASED
tobuster-wikimedia
.
- Remember to change the distribution from
- <WIP>
Troubleshooting
Entry in database_resolve that maps to a bare hostname
+--------------------+--------------------+---------------------+ | hostname | resolved_hostname | resolved_timestamp | +--------------------+--------------------+---------------------+ | pc1008.eqiad.wmnet | pc1008 | 2020-11-18 10:11:58 | +--------------------+--------------------+---------------------+
This can cause a 'ghost' cluster to appear, containing the bare-hostname version of the host. To fix this:
systemctl stop orchestrator orchestrator -c forget -i <instance> for all instances in the ghost cluster orchestrator -c reset-hostname-resolve-cache systemctl start orchestrator
Stopping orchestrator is required to stop it from reinserting the bad entry into hostname_resolve.
The entries can be queried via orchestrator -c show-resolve-hosts