You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Orchestrator

From Wikitech-static
Revision as of 11:07, 18 November 2020 by imported>Marostegui (Quick draft of troubleshooting section, this needs more work)
Jump to navigation Jump to search

Orchestrator is a service for managing mysql cluster replication. The data-persistence SRE team is currently doing a proof-of-concept deployment of it within WMF, with the aim of replacing Tendril/Dbtree.

Troubleshooting

Entry in database_resolve that maps to a bare hostname

+--------------------+--------------------+---------------------+
| hostname           | resolved_hostname  | resolved_timestamp  |
+--------------------+--------------------+---------------------+
| pc1008.eqiad.wmnet | pc1008             | 2020-11-18 10:11:58 |
+--------------------+--------------------+---------------------+

This can cause a 'ghost' cluster to appear, containing the bare-hostname version of the host. To fix this:

systemctl stop orchestrator
orchestrator -c forget -i <instance> for all instances in the ghost cluster
orchestrator -c reset-hostname-resolve-cache
systemctl start orchestrator

Stopping orchestrator is required to stop it from reinserting the bad entry into hostname_resolve.

The entries can be queried via orchestrator -c show-resolve-hosts