Cumin

Features

For a general description of Cumin's features, see https://github.com/wikimedia/cumin/blob/master/README.md

The TL;DR quick summary of Cumin features, relevant to the usage inside WMF are:

• Select target hosts by name and querying PuppetDB for any included Puppet Resource or Puppet Fact.
• Execute any number of arbitrary commands via SSH on the selected target hosts in an orchestrated way (see below) grouping the hosts the have the same output

Host selection

DISCLAIMER: the grammar used by Cumin to query its backends for the hosts selection will be modified in the near future to improve the capabilities of host selection. In particular it will be expanded to encapsulate the current grammar into composable blocks to allow more complex and powerful host selection queries. This documentation will be updated accordingly.

• Match hosts by name with a simple globbing:
  • wdqs2* matches all the hosts with hostname starting with wdqs2 hence all the Wikidata Query Service hosts in codfw. wdqs2*.codfw.wmnet is a more formal way to specify it.
  • wdqs2* or pc2* matches all the above plus the codfw's Parser Cache hosts.
• Match hosts using the ClusterShell NodeSet syntax:
  • db[2016-2019,2023,2028-2029,2033].codfw.wmnet define a specific list of hosts
• Puppet Fact selection:
  • F:memorysize_mb ~ "^[2-3][0-9][0-9][0-9][0-9]" selects all the hosts that have beween 20000MB and 39999MB of RAM
  • F:lsbdistid = Ubuntu and analytics* selects all the hosts with hostname that starts with analytics that have Ubuntu as OS
• Puppet Resource selection:
  • R:File = /etc/ssl/localcerts/api.svc.eqiad.wmnet.chained.crt selects all the hosts in which Puppet manages this specific file resource
  • R:Class = Mediawiki::Nutcracker and *.eqiad.wmnet selects all the hosts that have the Puppet Class Mediawiki::Nutcracker applied and the hostname ending in .eqiad.wmnet, that is a quick hack to select a single datacenter if there are not hosts .wikimedia.org involved until we'll expose $::site and other global variables to PuppetDB.
• Special all hosts matching: * !!!ATTENTION: use extreme caution with this selector!!!

Command execution

TODO

WMF installation

Production infrastructure

In the WMF production infrastructure, Cumin masters are installed via Puppet's Role::Cumin::Master role, that is currently included in the Role::Cluster::Management role. Cumin can be executed in any of those hosts and requires sudo privileges or being root. Cumin can access any production host that includes the Profile::Cumin::Target profile as root (all production hosts as of now), hence is a very powerful but also a potentially very dangerous tool, be very careful while using it. The current Cumin's masters from where it can be executed are:

Cumin master hosts
neodymium.eqiad.wmnet
sarin.codfw.wmnet

The default Cumin backend is configured to be PuppetDB and the default transport ClusterShell (SSH). The capability of Cumin to query PuppetDB as a backend allow to select hosts in a very powerful and precise way, querying for any Puppet resource or fact. Mixed query for resources and facts are currently not supported, but will be addressed by the grammar improvements described above.

Examples of usage in the WMF infrastructure

• Check semi-sync replication status (number of connected clients) on all core mediawiki master databases:

sudo cumin 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\""