Cumin: Difference between revisions

Latest revision as of 22:10, 5 June 2023

Cumin is a cluster automation and orchestration framework written in Python.

Resources

For a general description of Cumin's features, see the documentation introduction.
For a complete list of the latest changes, see the release notes page.
For an introductory video see the talk made at FOSDEM 2018: Cumin: Flexible and Reliable Automation for the Fleet.

Features

The TL;DR quick summary of Cumin features, relevant to the usage inside WMF are:

Select target hosts in the production environment by hostname and/or querying PuppetDB for any applied Puppet Resource or Fact. Only one main Resource per PuppetDB query can be specified and it's not possible to mix resources and facts selection in the same query. To overcome this limit is possible to use the general grammar and combine multiple subqueries to achieve the same result. For the WMCS Cloud VPS environment the OpenStack API are used. Please note that when using multiple host selection queries from the CLI the entire set of queries should be enclosed in quotes, as shown in the examples below.
Execute any number of arbitrary commands via SSH on the selected target hosts in parallel in an orchestrated way (see below) grouping the output for the hosts that have the same output.
Can be used directly as a CLI or as a Python 3 library.
A more higher-level tool perform common automation and orchestration tasks inside WMF that expose also Cumin as a library is available, see Spicerack.

Host selection

Our production configuration uses PuppetDB as default backend, meaning that by default each host selection query is parsed as a PuppetDB query, and only if the parsing fails it will be re-parsed with the general grammar. This allows to use everyday queries without additional syntax, while leaving the full power of composing subqueries when needed. You can find a comprehensive list of aliases for the PuppetDB backend in /etc/cumin/aliases.yaml where cumin is installed or see the template that generates them in the operations/puppet repo.

For WMCS instead the default configuration uses OpenStack as the default backend.

When using the CLI, the --dry-run option is useful to just check which hosts matches the query without executing any command, if no commands are specified this option is enabled automatically.

Pleae note that when using multiple host selection queries from the CLI the entire set of queries should be enclosed in quotes, as shown in the examples below.

PuppetDB host selection

Match hosts by exact FDQN:
- einsteinium.wikimedia.org with it's FQDN
- einsteinium.wikimedia.org,neodymium.eqiad.wmnet comma-separated list of FQDNs
Match hosts by FQDN with a simple globbing:
- wdqs2* matches all the hosts with hostname starting with wdqs2 hence all the Wikidata Query Service hosts in codfw. wdqs2*.codfw.wmnet is a more formal way to specify it.
- wdqs2* or pc2* matches the same of the above plus the codfw's Parser Cache hosts; it's basically a sets union.
Match hosts by hostname using the ClusterShell NodeSet syntax:
- db[2016-2019,2023,2028-2029,2033].codfw.wmnet define a specific list of hosts in a compact format.
- cp[2001-2026].codfw.wmnet and cp[2021-2026].codfw.wmnet matches only 6 hosts, cp[2021-2026].codfw.wmnet, it's basically a sets intersection.

For example: if we set A=[2001-2026] and B=[2021-2026] what elements do we have in A that we also have in B? 2021,2022,2023,2024,2025 and 2026 are the elements that we have in A and B. This is called the intersection of A and B.

Puppet Fact selection:
- F:memorysize_mb >= 24000 and F:memorysize_mb <= 64000 selects all the hosts that have between 24000MB and 64000MB of RAM as exported by facter.
- F:filesystems ~ xfs selects all the hosts that have an XFS filesystem, while using ~ "^xfs$" would have been equivalent of = xfs.
- F:lsbdistid = Debian and analytics* selects all the hosts with hostname that starts with analytics that have Debian as OS.
- F:virtual = physical selects all physical hosts, whereas * and not F:virtual = physical will do the opposite.
Puppet Resource selection. Any host reachable by Cumin includes the profile::cumin::target Puppet class, to which some variable and tags were added in order to expose to PuppetDB the datacenter, the cluster and all the roles applied to each host. See it's usage in some of these examples:
- R:File = /etc/ssl/localcerts/api.svc.eqiad.wmnet.chained.crt selects all the hosts in which Puppet manages this specific file resource
- R:Service::Node selects all the hosts that have the Service::Node resource included, as it works for custom-defined resources too
- R:Class = Nginx selects all the hosts that have the Puppet Class Nginx applied.
- R:Class = mediawiki::web::prod_sites selects all the hosts that have the Puppet Class mediawiki::web::prod_sites applied.
- C:nginx and *.eqiad.wmnet uses the Class shortcut and selects all the hosts that have the Puppet Class Nginx applied and the hostname ending in .eqiad.wmnet, that is a quick hack to select a single datacenter if there are no hosts of the type .wikimedia.org involved.
- P:cumin::target%cluster = cache_upload and R:class%site = codfw allows to overcome the above limitation and selects all the hosts in the codfw datacenter that are part of the cache_upload cluster, using the shortcut for profiles P:.
- O:cache::upload or O:cache::text selects all the hosts that have either the role cache::upload or the role cache::text, using the shortcut for roles O:.
- R:git::clone%ensure=latest selects all the hosts that have the resource git::clone but only where parameter ensure is set to latest..
- P:cumin::target%site = codfw and (R:class@tag = role::cache::text or R:class@tag = role::cache::upload) this syntax allows to mix a selection over roles with specific sites and clusters.
- R:Class ~ "(?i)role::cache::(upload|text)" and *.ulsfo.wmnet selects all the cache upload and text hosts in ulsfo, the (?i) allow to perform the query in a case-insensitive mode (our implementation of PuppetDB uses PostgreSQL as a backend and the regex syntax is backend-dependent) without having to set uppercase the first letter of each class path.
- O:Mariadb::Groups%mysql_group = core and R:Class%mysql_role = slave selects all the hosts that have the R:Class = Role::Mariadb::Groups class with the parameter mysql_group with value core and the parameter mysql_role with value slave. Currently you cannot filter based on boolean parameters (T161545)
To mix in the same query Puppet Resources and Puppet Facts or combine multiple Puppet Resources or multiple Puppet Facts, use the Global Grammar host selection explained below.
Special all hosts matching: * !!!ATTENTION: use extreme caution with this selector!!!

OpenStack backend

project:deployment-prep: selects all the hosts in the deployment-prep (a.k.a. beta) project.
project:deployment-prep name:kafka: selects all the hosts in the deployment-prep project that have kafka in the name. OpenStack do a regex search on its side, so kafka here will match all akafka, kafkaz and akafkaz. If you want to match exactly kafka, you need to use name:^kafka$'.
project:deployment-prep name:"^deployment-kafka[0-9]+$": selects all the hosts in the deployment-prep project that matches the regex.
Additional key:value parameters can be added, separated by space, according to the OpenStack list-servers API. Valid options are 'reservation_id', 'name', 'status', 'image', 'flavor', 'ip', 'changes-since', 'all_tenants'.
Special all hosts in all projects matching: * !!!ATTENTION: use extreme caution with this selector!!!
To mix multiple selections the general grammar can be used: O{project:project1} or O{project:project2}

SSH Known Hosts files backend

cumin1001.eqiad.wmnet: Simple selection of a specific FQDN
mw13[15-22].eqiad.wmnet,mw2222.codfw.wmnet: ClusterShell syntax for hosts expansion and comma-separated multiple FQDNs
*.wikimedia.org: ClusterShell syntax for hosts globbing
mw13*.eqiad.wmnet or (mw22*.codfw.wmnet and not (mw2222* or mw2224*)): A complex selection
*.*: All FQDN hostnames, to avoid including also the short hostnames present in the known hosts files.

Direct backend

Using the direct backend each hostname is used verbatim, so it must be the FQDN of the host in order to work.

cumin1001.eqiad.wmnet: Simple selection of a specific FQDN
mw13[15-22].eqiad.wmnet,mw2222.codfw.wmnet: ClusterShell syntax for hosts expansion and comma-separated multiple FQDNs

HostFile backend (Enabled only in Cloud VPS)

This is a custom backend enabled through Cumin's plugin features only in Cloud VPS (labs-puppetmaster and other VPS cumin masters)
It replicates the functionality provided by clush's --hostfile or --machinefile option, and may be used to specify a path to a file containing a list of single hosts, node sets or node groups, separated by newlines.
The host selection query looks like: F{/home/user/hosts.list} and can be mixed with other backends using the general Cumin grammar.

Global grammar host selection

Backend query: anything inside I{}, where I is the backend identifier, is treated as a subquery to be parsed and executed with the chosen backend to gather its results. The available backend identifier are:
- P{}: PuppetDB backend
- O{}: OpenStack backend
- K{}: SSH Known hosts files backend
- D{}: Direct backend
- F{}: HostFile backend (available only in Cloud VPS)
Aliases: aliases are defined in /etc/cumin/aliases.yaml and the file is provisioned by Puppet. To use an alias in the query just use A:alias_name, where alias_name is the key in the aliases.yaml file. It will be replaced with its value before parsing the query. The alias replacement is recursive to allow nesting aliases.
Aggregation: the subqueries can be aggregated through the boolean operators and, or, and not, xor and with parentheses () for maximum flexibility.
Examples:
- (P{O:Ganeti} or P{O:Gerrit}) and P{F:is_virtual = true}: all hosts with the Ganeti or Gerrit Puppet role and the Puppet fact is_virtual with value true.
- P{O:Ganeti} and A:eqiad: all the hosts with the Ganeti Puppet role in the eqiad datacenter.
- P{C:Mediawiki::Nutcracker} and (P{host[10-20]*} or A:alias_name): all hosts with the Mediawiki::Nutcracker Puppet class and matching either a given alias alias_name or hostnames that start with host[10-20].
- O{project:deployment-prep} and not D{deployment-logstash02.deployment-prep.eqiad1.wikimedia.cloud,deployment-imagescaler01.deployment-prep.eqiad1.wikimedia.cloud}: all hosts in the deployment-prep Openstack project but not the two listed there explicitly by FQDN.

Host list manipulation

On all Cumin master hosts the ClusterShell CLI tool nodeset is installed to allow for an easy manipulation of the host list from the ClusterShell syntax to any arbitrary syntax. See man nodeset for more details. A typical example is to expand the host list: nodeset -e -S '\n' HOST_LIST (where the HOST_LIST is the list of hosts returned by Cumin or pipe cumin with --no-colors e.g.

$ sudo cumin --no-colors P:sretest 2> /dev/null | nodeset -e -S '\n'
sretest1001.eqiad.wmnet
sretest1002.eqiad.wmnet

Command execution

There are various options that allow to control how the command execution will be performed. Keep in mind that Cumin by default assumes that any command executed was successful if it has an exit status code of 0, a failure otherwise.

Success threshold (default: 100%): consider the current parallel execution a failure only if the percentage of success is below this threshold. Useful when running multiple commands and/or using the execution in batches. Take into account that during the execution of a single command, if no batches were specified, the command will be executed on all the hosts and the success threshold checked only at the end. By default Cumin expects a 100% of success, a single failure will consider the execution failed. The CLI option is -p 0-100, --success-percentage 0-100.
Execute in batches (default: no batches, no sleep): by default Cumin schedule the execution in parallel on all the selected hosts. It is possible to specify to execute instead in batches. The batch execution mode of Cumin is with a sliding window of size N with an optional sleep of S seconds between hosts, with this workflow:
- It starts executing on the first batch of N hosts
- As soon as one host finishes the execution, if the success threshold is still met, schedule the execution on the next host in S seconds.
- At most N hosts will be executing the commands in parallel and the success threshold is check at each host completion.
- The CLI options are -b BATCH_SIZE, --batch-size BATCH_SIZE and -s BATCH_SLEEP, --batch-sleep BATCH_SLEEP and their default values are the number of hosts for the size and 0 seconds for the sleep.

Mode of execution (no default): when executing multiple commands, Cumin requires to specify a mode of execution. In the CLI there are two available modes: sync and async. In the library, in addition to those two modes, one can specify also a custom one. The CLI option is -m {sync,async}, --mode {sync,async}.
- sync execution:
  - execute the first command in parallel on all hosts, also considering the batch and success threshold parameters.
  - at the end of the execution, if the success threshold is met, start with the execution of the second command, and so on.
  - This allows to ensure that the first command was executed successfully on all hosts before proceeding with the next. Typical usage is when orchestrating changes across a cluster.
- async execution:
  - execute all the commands in sequence in each host, independently from one to each other, also considering the batch and success threshold parameters.
  - The execution on any given host is interrupted at the first command that fails.
  - It is kinda equivalent to an execution with a single command of the form command1 && command 2 && ... command N.
Ignore exit codes: there are situations in which the exit status of an executed command is not important (like when debugging stuff with grep) and showing it as a failure just make the output harder to read. In those cases the -x, --ignore-exit-codes option can be used, that assumes that every command executed was successful. !!!ATTENTION: use caution with this selector!!!
Timeout (default unlimited): an optional timeout to be applied to the execution of each command in each host, by default Cumin doesn't timeout. The CLI option is -t TIMEOUT, --timeout TIMEOUT.
Global timeout (default unlimited): an optional global timeout to the whole execution with Cumin, by default Cumin doesn't timeout. The CLI option is --global-timeout GLOBAL_TIMEOUT.

Output handling

Cumin's output can be modified using those options. At the moment all those options can be used only when a single command is executed. This limitation will be fixed in a future release.

Formatted output: it's possible to tell Cumin to print the output of the executed commands in a more parsable way, using the -o {txt,json}, --output {txt,json} option. When using this option the separator _____FORMATTED_OUTPUT_____ will be printed after the normal Cumin output and after it the output of the executed commands will be printed in the desired format, for each host, the usual Cumin de-duplication of output does not apply to the formatted output. To just extract the formatted output you can append | awk 'x==1 { print $0 } /_____FORMATTED_OUTPUT_____/ { x=1 }' to the Cumin command in the general case. This limitation will be fixed in v5.0.0. If stderr is not needed you can combine various options to make the extraction even simpler, see the example below. The available formats are:
- txt: using this format will prepend the ${HOSTNAME}: to each line of output for that host, keeping the existing newlines.
- json: using this format will print a JSON dictionary where the keys are the hostnames and the value is a string with the whole output of the host
```
cumin --force --no-progress --no-color -o txt 'A:cumin' 'date' 2>/dev/null | tail -n "+2" | tee -a example.out
```

Interactive mode: if you want to manipulate the results with the power of Python, using the -i, --interactive option Cumin will drop into a Python REPL session at the end of the execution, having direct access to Cumin's objects for further processing.

WMF installation

Production infrastructure

In the WMF Production infrastructure, Cumin masters are installed via Puppet's Role::Cumin::Master role, that is currently included in the Role::Cluster::Management role. Cumin can be executed in any of those hosts and requires sudo privileges or being root. Cumin can access any production host that includes the Profile::Cumin::Target profile as root (all production hosts as of now), hence is a very powerful but also a potentially very dangerous tool, be very careful while using it. The current Cumin's masters from where it can be executed are:

Cumin master hosts in production
`cumin1001.eqiad.wmnet`
`cumin2002.codfw.wmnet`

The default Cumin backend is configured to be PuppetDB and the default transport ClusterShell (SSH). The capability of Cumin to query PuppetDB as a backend allow to select hosts in a very powerful and precise way, querying for any Puppet resources or facts.

If running commands on hosts only in one of the DC where there is a Cumin master consider running it from the local Cumin master to slightly speed up the execution.

Please note that the Openstack backend is currently not available for cumin masters in production.

Rootless production infrastructure

In addition to the Cumin masters available for users with global root, there's also a separate installation for rootless operation based on Kerberos, that gets installed via the cluster::managementunpriv role. Rootless cumin can only access production host which have been enabled for rootless Cumin, see below. The current rootless Cumin masters are:

Cumin master hosts in production
`cuminunpriv1001.eqiad.wmnet`

After logging in,you need to run "kinit" which activates your Kerberos ticket. Following that you can run commands on kerberos-enabled hosts (see below) as your standard user:

jmm@cuminunpriv1001:~$ cumin A:installserver 'uname -v'
7 hosts will be targeted:
apt[1001,2001].wikimedia.org,install[1003,2003,3001,4001,5001].wikimedia.org
Ok to proceed on 7 hosts? Enter the number of affected hosts to confirm or "q" to quit 7
===== NODE GROUP =====
(7) apt[1001,2001].wikimedia.org,install[1003,2003,3001,4001,5001].wikimedia.org
----- OUTPUT of 'uname -v' -----
#1 SMP Debian 4.19.171-2 (2021-01-30)
================
PASS |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (7/7) [00:03<00:00,  1.32hosts/s]
FAIL |                                                                                                                                                |   0% (0/7) [00:03<?, ?hosts/s]
100.0% (7/7) success ratio (>= 100.0% threshold) for command: 'uname -v'.
100.0% (7/7) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Making hosts manageable with rootless Cumin

Rootless cumin can only access production host which include the profile::base::cuminunpriv profile. In addition a host principal/keytab need to be created, following the docs at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos#Create_a_keytab_for_a_service (the example given there for sretest1001 creates the necessary keytab/principal).

Finally you need to make a Puppet change to the role of the service:

Include profile::base::cuminunpriv
Pass the following via Hiera:

profile::base::ssh_server_settings:
  enable_kerberos: true

WMCS Cloud VPS infrastructure

There are dedicated Cumin masters in the restricted cloudinfra project for Cloud VPS admin use. Cumin can access any Cloud VPS VMs (except special VMs managed by Trove). Hence is a very powerful but also a potentially very dangerous tool, be very careful while using it.

The current Cumin's masters from where it can be executed are:

Cumin master hosts in Cloud VPS
`cloud-cumin-03.cloudinfra.eqiad1.wikimedia.cloud`
`cloud-cumin-04.cloudinfra.eqiad1.wikimedia.cloud`

Please note that the PuppetDB backend is currently not available for the cloud-wide cumin masters in Cloud VPS.

WARNING: Be careful if you are specifying a batch size (-b) in Cumin. Values above 3 will probably cause timeouts while establishing the SSH connections.

WMCS Cloud VPS single project installation

Independently of the above global installations, Cumin can be also easily installed inside a Cloud VPS project, see the detailed instructions in Help:Cumin master.

Cumin CLI examples in the WMF production infrastructure

Run Puppet discarding the output

To run Puppet on a set of hosts without getting the output, just relying on the exit code, one host at the time, sleeping 5 seconds between one host and the next, proceeding to the next host only if the current one succeeded. Avoid puppet agent -t because that includes the --detailed-exitcodes option that returns exit codes > 0 also in successful cases:

$ sudo cumin -b 20 -s 2 'wdqs2*' 'run-puppet-agent -q'
3 hosts will be targeted:
wdqs[2001-2003].codfw.wmnet
Confirm to continue [y/n]? y
===== NO OUTPUT =====
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [02:24<00:00, 46.03s/hosts]
FAIL |                                                                                                         |   0% (0/3) [02:24<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command: 'run-puppet-agent -q'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Run Puppet keeping the output

$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent'

Disable Puppet

To disable Puppet in a consistent way, waiting for the completion of any in flight puppet runs:

$ sudo cumin 'wdqs2*' "disable-puppet 'Reason why was disabled - T12345 - ${USER}'"

Enable Puppet

To enable Puppet only on the hosts where was disabled with the same message:

$ sudo cumin 'wdqs2*' "enable-puppet 'Reason why was disabled - T12345 - ${USER}'"

Run Puppet only if last run failed

Might happen that a change merged in Puppet causes Puppet to fail on a number of hosts. Once the issue is fixed, without the need to wait for the next Puppet run, an easy way to quickly fix Puppet on all the failed hosts is to run the following command. It will exit immediately if the last puppet run was successful and run puppet only on the host where it failed and is of course enabled. the -p 95 option is to take into account that some hosts might be down/unreachable without making cumin fail. Remove the -q if you want to get the output, although might be very verbose based on the number of hosts that failed last run:

sudo cumin -b 20 -p 95 '*' 'run-puppet-agent -q --failed-only'

Check if systemd service is running

$ sudo cumin 'A:mw-api and A:codfw' 'systemctl is-active php7.2-fpm.service'
64 hosts will be targeted:
mw[2251-2253,2261-2262,2283-2300,2302,2304,2306,2308,2317-2324,2326,2328,2330,2332,2334,2350,2352,2354,2356,2358,2360,2362,2364,2366,2368,2370,2372,2374,2376,2396-2405].codfw.wmnet
Ok to proceed on 64 hosts? Enter the number of affected hosts to confirm or "q" to quit 64
===== NODE GROUP =====
(64) mw[2251-2253,2261-2262,2283-2300,2302,2304,2306,2308,2317-2324,2326,2328,2330,2332,2334,2350,2352,2354,2356,2358,2360,2362,2364,2366,2368,2370,2372,2374,2376,2396-2405].codfw.wmnet
----- OUTPUT of 'systemctl is-act...p7.2-fpm.service' -----
active
================
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (64/64) [00:00<00:00, 95.26hosts/s]
FAIL |                                                                                                                      |   0% (0/64) [00:00<?, ?hosts/s]
100.0% (64/64) success ratio (>= 100.0% threshold) for command: 'systemctl is-act...p7.2-fpm.service'.
100.0% (64/64) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Reboot

Given that Cumin uses SSH as a transport, just running reboot will most likely leave the connection hanging and it will not properly return to Cumin. To overcome this, use the reboot-host command (which nohups the reboot command in a wrapper) which we ship via puppet:

$ cumin <hosts> 'reboot-host'

Check TLS certificate

Print a TLS certificate from all the hosts that have that specific Puppet-managed file to ensure that is the same on all hosts and to verify its details. The expected output in case all the hosts have the same certificate is only one block with the certificate content with the number and list of the hosts that have it on top:

$ sudo cumin 'R:File = /etc/ssl/localcerts/api.svc.codfw.wmnet.chained.crt' 'openssl x509 -in /etc/ssl/localcerts/api.svc.codfw.wmnet.chained.crt -text -noout'

Check TLS private key

Ensure that the private key of a certificate matches the certificate itself on all the hosts that have a specific certificate, can be done in two ways:

Using the async mode only one line of output is expected, the matching MD5 for all the hosts for both the certificate and the private key.
Using the sync mode instead 2 lines of grouped output are expected, one for the first command and one for the second one, leaving the user to match those.

$ sudo cumin -m async 'R:File = /etc/ssl/localcerts/api.svc.codfw.wmnet.chained.crt' 'openssl pkey -pubout -in /etc/ssl/private/api.svc.codfw.wmnet.key | openssl md5' 'openssl x509 -pubkey -in /etc/ssl/localcerts/api.svc.codfw.wmnet.chained.crt -noout | openssl md5'
55 hosts will be targeted:
mw[2120-2147,2200-2223,2251-2253].codfw.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(110) mw[2120-2147,2200-2223,2251-2253].codfw.wmnet
----- OUTPUT -----
(stdin)= c51627f0b52a4dc70d693acdfdf4384a
================
PASS |████████████████████████████████████████████████████████████████████████████████████████████████| 100% (55/55) [00:00<00:00, 89.83hosts/s]
FAIL |                                                                                                         |   0% (0/55) [00:00<?, ?hosts/s]
100.0% (55/55) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Check MySQL semi-sync replication status

Check semi-sync replication status (number of connected clients) on all core mediawiki master databases:

$ sudo cumin 'O:Mariadb::Groups%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\""

Upgrade Debian packages

From time to time it is necessary to roll out new versions of Debian packages. If the version isn't explicitly stated in puppet with ensure then the package won't be normally upgraded.

NB: Make sure to test the upgrade on a few selected nodes first.

NB2: Even better, use debdeploy in production to run package upgrades.

cumin 'HOSTS' 'DEBIAN_FRONTEND=noninteractive apt-get -q -y --assume-no -o \
  DPkg::Options::="--force-confdef" -o DPkg::Options::="--force-confold" \
  install PACKAGE'

Check EtcdConfig for MediaWiki hosts

Check that the part of mediawiki-config that comes from etcd, and is exposed as wmf-config in siteinfo is in sync with etcd checking that the lastModifiedIndex is the last one. The lastModifiedIndex is expected to be different in the two etcd clusters (eqiad vs codfw).

$ sudo cumin 'A:mw or A:mw-api' "curl -sx \$(hostname -f):80 -H'X-Forwarded-Proto: https' 'http://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&format=json&formatversion=2' | jq -r '.query.general[\"wmf-config\"].wmfEtcdLastModifiedIndex'"

To check the actual content of the wmf-config section of the configuration, just remove the .wmfEtcdLastModifiedIndex at the end.

Target all hosts running Stretch

$ sudo cumin 'A:stretch'

Troubleshooting Production issues

Logs

Cumin's log are in /var/log/cumin

PuppetDB is down

If PuppetDB is not working for some reason (host down, software problems, etc.) Cumin will fail to match hosts based on compound expressions. The Known Hosts and Direct backends will still work using the global grammar syntax with K{} (see Cumin#SSH Known Hosts files backend) and D{} respectively. Alternatively the --backend knownhosts and --backend direct option can also be used respectively. It might make sense in any case to fallback to the secondary PuppetDB host. For a quick fix disable puppet and edit /etc/cumin/config.yaml, in the puppetdb section, amend the:

   host: puppetdb1001.eqiad.wmnet

to be:

   host: puppetdb2001.codfw.wmnet

and try to see if cumin is working again. For a permanent fix adjust the hiera variable profile::cumin::master::puppetdb_host.

Sudo is broken

If sudo is broken, first of all it can be fixed in Puppet, as Puppet runs as root and doesn't need sudo.

But you might need Cumin to perform a quick fix before Puppet in oder to reduce the disruption. To overcome the problem that to run Cumin you need to be root follow these steps:

SSH to one of the Cumin hosts (cumin1001.eqiad.wmnet, cumin2002.codfw.wmnet)
- In the rare case that is not possible to SSH to the Cumin hosts, is still possible to get a root's shell on the Cumin host SSHing to the management console and connecting to the serial console
From the Cumin host, SSH as root to an affected host using the Cumin SSH key: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh root@$FQDN
Test on the affected host the commands required to fix the host so that they can be later run via Cumin on all affected hosts
Once a safe command to be run via Cumin has been found, get a root shell on the Cumin host using su and the root's password present in pwstore
Run cumin as usual from the root's shell without sudo

How to contribute to Cumin

Did you find a bug?

Check if it was already reported or resolved looking at Cumin All Issues on Wikimedia Phabricator. Open issues are listed at Cumin Open Issues.

If you're unable to find a related issue, open a new one in Phabricator:
- For WMF Production and WMCS Cloud VPS global installations, use this template.
- For any Cumin usage outside WMF infrastructure and for WMCS Cloud VPS single project installations, use this template.

Contributing to the code

Cumin's development is done on Wikimedia's Gerrit. A developer account is needed to send a pull request.
If it's a patch to fix a bug, make sure that a bug was already reported and confirmed, see the above section.
if it's a patch for a new feature, make sure to open an issue first and that there is consensus to add this new feature to Cumin.
Check the Development page in Cumin's documentation to have a quick overview of its structure.
Make sure that all the tests are passing, including the integration tests, if possible.
Make sure to reference the existing open issue(s) related to the patch adding a line with Bug: T00001 as the last line of your commit message (replacing the issue ID with the correct one(s)). Follow mw:Gerrit/Commit message guidelines.
If you need help, feel free to contact mw:User:RCoccioli_(WMF).