You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Automation and orchestration framework written in Python
For a general description of Cumin's features, see https://github.com/wikimedia/cumin/blob/master/README.md
The TL;DR quick summary of Cumin features, relevant to the usage inside WMF are:
- Select target hosts by name and querying PuppetDB for any included Puppet Resource or Puppet Fact.
- Execute any number of arbitrary commands via SSH on the selected target hosts in an orchestrated way (see below) grouping the hosts the have the same output
DISCLAIMER: the grammar used by Cumin to query its backends for the hosts selection will be modified in the near future to improve the capabilities of host selection. In particular it will be expanded to encapsulate the current grammar into composable blocks to allow more complex and powerful host selection queries. This documentation will be updated accordingly.
- Match hosts by name with a simple globbing:
wdqs2*matches all the hosts with hostname starting with
wdqs2hence all the Wikidata Query Service hosts in codfw.
wdqs2*.codfw.wmnetis a more formal way to specify it.
wdqs2* or pc2*matches all the above plus the codfw's Parser Cache hosts.
- Match hosts using the ClusterShell NodeSet syntax:
db[2016-2019,2023,2028-2029,2033].codfw.wmnetdefine a specific list of hosts
- Puppet Fact selection:
F:memorysize_mb ~ "^[2-3][0-9][0-9][0-9][0-9]"selects all the hosts that have beween 20000MB and 39999MB of RAM
F:lsbdistid = Ubuntu and analytics*selects all the hosts with hostname that starts with
analyticsthat have Ubuntu as OS
- Puppet Resource selection:
R:File = /etc/ssl/localcerts/api.svc.eqiad.wmnet.chained.crtselects all the hosts in which Puppet manages this specific file resource
R:Class = Mediawiki::Nutcracker and *.eqiad.wmnetselects all the hosts that have the Puppet Class
Mediawiki::Nutcrackerapplied and the hostname ending in
.eqiad.wmnet, that is a quick hack to select a single datacenter if there are not hosts
.wikimedia.orginvolved until we'll expose
$::siteand other global variables to PuppetDB.
- Special all hosts matching:
*!!!ATTENTION: use extreme caution with this selector!!!
In the WMF production infrastructure, Cumin masters are installed via Puppet's
Role::Cumin::Master role, that is currently included in the
Role::Cluster::Management role. Cumin can be executed in any of those hosts and requires sudo privileges or being root. Cumin can access any production host that includes the
Profile::Cumin::Target profile as root (all production hosts as of now), hence is a very powerful but also a potentially very dangerous tool, be very careful while using it. The current Cumin's masters from where it can be executed are:
|Cumin master hosts|
The default Cumin backend is configured to be PuppetDB and the default transport ClusterShell (SSH). The capability of Cumin to query PuppetDB as a backend allow to select hosts in a very powerful and precise way, querying for any Puppet resource or fact. Mixed query for resources and facts are currently not supported, but will be addressed by the grammar improvements described above.
Examples of usage in the WMF infrastructure
- Check semi-sync replication status (number of connected clients) on all core mediawiki master databases:
sudo cumin 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\""