You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Cumin

From Wikitech-static
Revision as of 16:38, 21 April 2020 by imported>Volans (→‎Production infrastructure: removed decommissioned hosts)
Jump to navigation Jump to search

Automation and orchestration framework written in Python

Resources

Features

The TL;DR quick summary of Cumin features, relevant to the usage inside WMF are:

  • Select target hosts in the production environment by hostname and/or querying PuppetDB for any applied Puppet Resource or Fact. Only one main Resource per PuppetDB query can be specified and it's not possible to mix resources and facts selection in the same query. To overcome this limit is possible to use the general grammar and combine multiple subqueries to achieve the same result. For the WMCS Cloud VPS environment the OpenStack API are used.
  • Execute any number of arbitrary commands via SSH on the selected target hosts in parallel in an orchestrated way (see below) grouping the output for the hosts that have the same output.
  • Can be used directly as a CLI or as a Python 2 library (Python 3 support will be added soon, as the latest dependency that was Python 2 only has recently added support for Python 3).
  • In the near future a more higher-level tool will be developed that will use Cumin and other libraries to perform common automation and orchestration tasks inside WMF.

Host selection

Our production configuration uses PuppetDB as default backend, meaning that by default each host selection query is parsed as a PuppetDB query, and only if the parsing fails it will be re-parsed with the general grammar. This allows to use everyday queries without additional syntax, while leaving the full power of composing subqueries when needed.

For WMCS instead the default configuration uses OpenStack as the default backend.

When using the CLI, the --dry-run option is useful to just check which hosts matches the query without executing any command, if no commands are specified this option is enabled automatically.

PuppetDB host selection

  • Match hosts by exact FDQN:
    • einsteinium.wikimedia.org with it's FQDN
    • einsteinium.wikimedia.org,neodymium.eqiad.wmnet comma-separated list of FQDNs
  • Match hosts by FQDN with a simple globbing:
    • wdqs2* matches all the hosts with hostname starting with wdqs2 hence all the Wikidata Query Service hosts in codfw. wdqs2*.codfw.wmnet is a more formal way to specify it.
    • wdqs2* or pc2* matches the same of the above plus the codfw's Parser Cache hosts, it's basically a sets union.
  • Match hosts by hostname using the ClusterShell NodeSet syntax:
    • db[2016-2019,2023,2028-2029,2033].codfw.wmnet define a specific list of hosts in a compact format.
    • cp[2001-2026].codfw.wmnet and cp[2021-2026].codfw.wmnet matches only 6 hosts, cp[2021-2026].codfw.wmnet, it's basically a sets intersection.

For example: if we set A=[2001-2026] and B=[2021-2026] what elements do we have in A that we also have in B? 2021,2022,2023,2024,2025 and 2026 are the elements that we have in A and B. This is called the intersection of A and B.

  • Puppet Fact selection:
    • F:memorysize_mb >= 24000 and F:memorysize_mb <= 64000 selects all the hosts that have between 24000MB and 64000MB of RAM as exported by facter.
    • F:filesystems ~ xfs selects all the hosts that have an XFS filesystem, while using ~ "^xfs$" would have been equivalent of = xfs.
    • F:lsbdistid = Debian and analytics* selects all the hosts with hostname that starts with analytics that have Debian as OS.
    • F:virtual = physical selects all physical hosts, whereas * and not F:virtual = physical will do the opposite.
  • Puppet Resource selection. Any host reachable by Cumin includes the profile::cumin::target Puppet class, to which some variable and tags were added in order to expose to PuppetDB the datacenter, the cluster and all the roles applied to each host. See it's usage in some of these examples:
    • R:File = /etc/ssl/localcerts/api.svc.eqiad.wmnet.chained.crt selects all the hosts in which Puppet manages this specific file resource
    • R:Service::Node selects all the hosts that have the Service::Node resource included, as it works for custom-defined resources too
    • R:Class = Nginx selects all the hosts that have the Puppet Class Nginx applied.
    • R:Class = mediawiki::web::prod_sites selects all the hosts that have the Puppet Class mediawiki::web::prod_sites applied.
    • C:nginx and *.eqiad.wmnet uses the Class shortcut and selects all the hosts that have the Puppet Class Nginx applied and the hostname ending in .eqiad.wmnet, that is a quick hack to select a single datacenter if there are no hosts of the type .wikimedia.org involved.
    • P:cumin::target%cluster = cache_upload and R:class%site = codfw allows to overcome the above limitation and selects all the hosts in the codfw datacenter that are part of the cache_upload cluster, using the shortcut for profiles P:.
    • O:cache::upload or O:cache::text selects all the hosts that have either the role cache::upload or the role cache::text, using the shortcut for roles O:.
    • P:cumin::target%site = codfw and (R:class@tag = role::cache::text or R:class@tag = role::cache::upload) this syntax allows to mix a selection over roles with specific sites and clusters.
    • R:Class ~ "(?i)role::cache::(upload|text)" and *.ulsfo.wmnet selects all the cache upload and text hosts in ulsfo, the (?i) allow to perform the query in a case-insensitive mode (our implementation of PuppetDB uses PostgreSQL as a backend and the regex syntax is backend-dependent) without having to set uppercase the first letter of each class path.
    • O:Mariadb::Groups%mysql_group = core and R:Class%mysql_role = slave selects all the hosts that have the R:Class = Role::Mariadb::Groups class with the parameter mysql_group with value core and the parameter mysql_role with value slave. Currently you cannot filter based on boolean parameters (T161545)
  • Special all hosts matching: * !!!ATTENTION: use extreme caution with this selector!!!

OpenStack backend

  • project:deployment-prep: selects all the hosts in the deployment-prep (a.k.a. beta) project.
  • project:deployment-prep name:kafka: selects all the hosts in the deployment-prep project that have kafka in the name. OpenStack do a regex search on its side, so kafka here will match all akafka, kafkaz and akafkaz. If you want to match exactly kafka, you need to use name:^kafka$'.
  • project:deployment-prep name:"^deployment-kafka[0-9]+$": selects all the hosts in the deployment-prep project that matches the regex.
  • Additional key:value parameters can be added, separated by space, according to the OpenStack list-servers API. Valid options are 'reservation_id', 'name', 'status', 'image', 'flavor', 'ip', 'changes-since', 'all_tenants'.
  • Special all hosts in all projects matching: * !!!ATTENTION: use extreme caution with this selector!!!
  • To mix multiple selections the general grammar can be used: O{project:project1} or O{project:project2}

SSH Known Hosts files backend

  • cumin1001.eqiad.wmnet: Simple selection of a specific FQDN
  • mw13[15-22].eqiad.wmnet,mw2222.codfw.wmnet: ClusterShell syntax for hosts expansion and comma-separated multiple FQDNs
  • *.wikimedia.org: ClusterShell syntax for hosts globbing
  • mw13*.eqiad.wmnet or (mw22*.codfw.wmnet and not (mw2222* or mw2224*)): A complex selection
  • *.*: All FQDN hostnames, to avoid including also the short hostnames present in the known hosts files.

HostFile backend (Enabled only in Cloud VPS)

  • This is a custom backend enabled through Cumin's plugin features only in Cloud VPS (labs-puppetmaster and other VPS cumin masters)
  • It replicates the functionality provided by clush's --hostfile or --machinefile option, and may be used to specify a path to a file containing a list of single hosts, node sets or node groups, separated by newlines.
  • The host selection query looks like: F{/home/user/hosts.list} and can be mixed with other backends using the general Cumin grammar.

Global grammar host selection

  • Backend query: anything inside I{}, where I is the backend identifier, is treated as a subquery to be parsed and executed with the chosen backend to gather its results. The available backend identifier are:
    • P{}: PuppetDB backend
    • O{}: OpenStack backend
    • K{}: SSH Known hosts files backend
    • D{}: Direct backend
    • F{}: HostFile backend (available only in Cloud VPS)
  • Aliases: aliases are defined in /etc/cumin/aliases.yaml and the file is provisioned by Puppet. To use an alias in the query just use A:alias_name, where alias_name is the key in the aliases.yaml file. It will be replaced with its value before parsing the query. The alias replacement is recursive to allow nesting aliases.
  • Aggregation: the subqueries can be aggregated through the boolean operators and, or, and not, xor and with parentheses ()for maximum flexibility.
  • Example: P{C:Mediawiki::Nutcracker} and (D{host[10-20]} or A:alias_name)

Host list manipulation

On all Cumin master hosts the ClusterShell CLI tool nodeset is installed to allow for an easy manipulation of the host list from the ClusterShell syntax to any arbitrary syntax. See man nodeset for more details. A typical example is to expand the host list: nodeset -e -S '\n' HOST_LIST (where the HOST_LIST is the list of hosts returned by Cumin.

Command execution

There are various options that allow to control how the command execution will be performed. Keep in mind that Cumin by default assumes that any command executed was successful if it has an exit status code of 0, a failure otherwise.

  • Success threshold (default: 100%): consider the current parallel execution a failure only if the percentage of success is below this threshold. Useful when running multiple commands and/or using the execution in batches. Take into account that during the execution of a single command, if no batches were specified, the command will be executed on all the hosts and the success threshold checked only at the end. By default Cumin expects a 100% of success, a single failure will consider the execution failed. The CLI option is -p 0-100, --success-percentage 0-100.
  • Execute in batches (default: no batches, no sleep): by default Cumin schedule the execution in parallel on all the selected hosts. It is possible to specify to execute instead in batches. The batch execution mode of Cumin is with a sliding window of size N with an optional sleep of S seconds between hosts, with this workflow:
    • It starts executing on the first batch of N hosts
    • As soon as one host finishes the execution, if the success threshold is still met, schedule the execution on the next host in S seconds.
    • At most N hosts will be executing the commands in parallel and the success threshold is check at each host completion.
    • The CLI options are -b BATCH_SIZE, --batch-size BATCH_SIZE and -s BATCH_SLEEP, --batch-sleep BATCH_SLEEP and their default values are the number of hosts for the size and 0 seconds for the sleep.
  • Mode of execution (no default): when executing multiple commands, Cumin requires to specify a mode of execution. In the CLI there are two available modes: sync and async. In the library, in addition to those two modes, one can specify also a custom one. The CLI option is -m {sync,async}, --mode {sync,async}.
    • sync execution:
      • execute the first command in parallel on all hosts, also considering the batch and success threshold parameters.
      • at the end of the execution, if the success threshold is met, start with the execution of the second command, and so on.
      • This allows to ensure that the first command was executed successfully on all hosts before proceeding with the next. Typical usage is when orchestrating changes across a cluster.
    • async execution:
      • execute all the commands in sequence in each host, independently from one to each other, also considering the batch and success threshold parameters.
      • The execution on any given host is interrupted at the first command that fails.
      • It is kinda equivalent to an execution with a single command of the form command1 && command 2 && ... command N.
  • Ignore exit codes: there are situations in which the exit status of an executed command is not important (like when debugging stuff with grep) and showing it as a failure just make the output harder to read. In those cases the -x, --ignore-exit-codes option can be used, that assumes that every command executed was successful. !!!ATTENTION: use caution with this selector!!!
  • Timeout (default unlimited): an optional timeout to be applied to the execution of each command in each host, by default Cumin doesn't timeout. The CLI option is -t TIMEOUT, --timeout TIMEOUT.
  • Global timeout (default unlimited): an optional global timeout to the whole execution with Cumin, by default Cumin doesn't timeout. The CLI option is --global-timeout GLOBAL_TIMEOUT.

Output handling

Cumin's output can be modified using those options. At the moment all those options can be used only when a single command is executed. This limitation will be fixed in a future release.

  • Formatted output: it's possible to tell Cumin to print the output of the executed commands in a more parsable way, using the -o {txt,json}, --output {txt,json} option. When using this option the separator _____FORMATTED_OUTPUT_____ will be printed after the normal Cumin output and after it the output of the executed commands will be printed in the desired format, for each host, the usual Cumin de-duplication of output does not apply to the formatted output. To just extract the formatted output append 2> /dev/null | awk 'x==1 { print $0 } /_____FORMATTED_OUTPUT_____/ { x=1 }' to the Cumin command. This limitation will be fixed in a future release. If you want to keep the stderr output just skip the /dev/null redirection. The available formats are:
    • txt: using this format will prepend the ${HOSTNAME}: to each line of output for that host, keeping the existing newlines.
      cumin --force -o txt 'A:cumin' 'date' | awk 'x==1 { print $0 } /_____FORMATTED_OUTPUT_____/ { x=1 }' | tee -a example.out
    • json: using this format will print a JSON dictionary where the keys are the hostnames and the value is a string with the whole output of the host.
  • Interactive mode: if you want to manipulate the results with the power of Python, using the -i, --interactive option Cumin will drop into a Python REPL session at the end of the execution, having direct access to Cumin's objects for further processing.

WMF installation

Production infrastructure

In the WMF Production infrastructure, Cumin masters are installed via Puppet's Role::Cumin::Master role, that is currently included in the Role::Cluster::Management role. Cumin can be executed in any of those hosts and requires sudo privileges or being root. Cumin can access any production host that includes the Profile::Cumin::Target profile as root (all production hosts as of now), hence is a very powerful but also a potentially very dangerous tool, be very careful while using it. The current Cumin's masters from where it can be executed are:

Cumin master hosts in production
cumin1001.eqiad.wmnet
cumin2001.codfw.wmnet

The default Cumin backend is configured to be PuppetDB and the default transport ClusterShell (SSH). The capability of Cumin to query PuppetDB as a backend allow to select hosts in a very powerful and precise way, querying for any Puppet resources or facts.

If running commands on hosts only in one of the DC where there is a Cumin master consider running it from the local Cumin master to slightly speed up the execution.

Please note that the Openstack backend is currently not available for cumin masters in production.

WMCS Cloud VPS infrastructure

In the WMCS infrastructure, Cumin masters are installed via Puppet's Profile::Openstack::eqiad1::Cumin::Master profile, that is currently included Role::Labs::Puppetmaster::Frontend and Role::Labs::Puppetmaster::Frontend roles. Cumin can be executed in any of those hosts and requires sudo privileges or being root. Cumin can access any Cloud VPS that includes the Profile::Openstack::eqiad1::Cumin::Target profile as root (all Cloud VPS as of now), hence is a very powerful but also a potentially very dangerous tool, be very careful while using it. The current Cumin's masters from where it can be executed are:

Cumin master hosts in WMCS Cloud VPS
cloud-cumin-01.cloudinfra.eqiad.wmflabs
cloud-cumin-02.cloudinfra.eqiad.wmflabs

Please note that the PuppetDB backend is currently not available for cumin masters in Cloud VPS.

WARNING: Be careful if you are specifying a batch size (-b) in Cumin. Values above 3 will probably cause timeouts while establishing the SSH connections.

WMCS Cloud VPS single project installation

Independently of the above global installations, Cumin can be also easily installed inside a Cloud VPS project, see the detailed instructions in Help:Cumin master.

Cumin CLI examples in the WMF production infrastructure

Run Puppet discarding the output

To run Puppet on a set of hosts without getting the output, just relying on the exit code, one host at the time, sleeping 5 seconds between one host and the next, proceeding to the next host only if the current one succeeded. Do not use puppet agent -t because that includes the --detailed-exitcodes option that returns exit codes > 0 also in successful cases:

$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent -q'
3 hosts will be targeted:
wdqs[2001-2003].codfw.wmnet
Confirm to continue [y/n]? y
===== NO OUTPUT =====
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [02:24<00:00, 46.03s/hosts]
FAIL |                                                                                                         |   0% (0/3) [02:24<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command: 'run-puppet-agent -q'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Run Puppet keeping the output

$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent'

Disable Puppet

To disable Puppet in a consistent way, waiting for the completion of any in flight puppet runs:

$ sudo cumin 'wdqs2*' "disable-puppet 'Reason why was disabled - T12345 - ${USER}'"

Enable Puppet

To enable Puppet only on the hosts where was disabled with the same message:

$ sudo cumin 'wdqs2*' "enable-puppet 'Reason why was disabled - T12345 - ${USER}'"

Run Puppet only if last run failed

Might happen that a change merged in Puppet causes Puppet to fail on a number of hosts. Once the issue is fixed, without the need to wait for the next Puppet run, an easy way to quickly fix Puppet on all the failed hosts is to run the following command. It will exit immediately if the last puppet run was successful and run puppet only on the host where it failed and is of course enabled. the -p 95 option is to take into account that some hosts might be down/unreachable without making cumin fail. Remove the -q if you want to get the output, although might be very verbose based on the number of hosts that failed last run:

sudo cumin -b 15 -p 95 '*' 'run-puppet-agent -q --failed-only'

Check if systemd service is running

$ sudo cumin 'P{O:mediawiki::appserver::api} and A:codfw' 'systemctl is-active hhvm.service'
55 hosts will be targeted:
mw[2120-2147,2200-2223,2251-2253].codfw.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(55) mw[2120-2147,2200-2223,2251-2253].codfw.wmnet
----- OUTPUT of 'systemctl is-active hhvm.service' -----
active
================
PASS |███████████████████████████████████████████████████████████████████████████████████████████████| 100% (55/55) [00:00<00:00, 148.77hosts/s]
FAIL |                                                                                                         |   0% (0/55) [00:00<?, ?hosts/s]
100.0% (55/55) success ratio (>= 100.0% threshold) for command: 'systemctl is-active hhvm.service'.
100.0% (55/55) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Reboot

Given that Cumin uses SSH as a transport, just running reboot will most likely left the connection hanging and it will not properly return to Cumin. To overcome this, in order to issue a proper reboot through Cumin (or SSH in general for that matters), use the reboot-host command (which nohups the reboot command in a wrapper) which we ship via puppet:

'reboot-host'

Check TLS certificate

Print a TLS certificate from all the hosts that have that specific Puppet-managed file to ensure that is the same on all hosts and to verify its details. The expected output in case all the hosts have the same certificate is only one block with the certificate content with the number and list of the hosts that have it on top:

$ sudo cumin 'R:File = /etc/ssl/localcerts/api.svc.codfw.wmnet.chained.crt' 'openssl x509 -in /etc/ssl/localcerts/api.svc.codfw.wmnet.chained.crt -text -noout'

Check TLS private key

Ensure that the private key of a certificate matches the certificate itself on all the hosts that have a specific certificate, can be done in two ways:

  • Using the async mode only one line of output is expected, the matching MD5 for all the hosts for both the certificate and the private key.
  • Using the sync mode instead 2 lines of grouped output are expected, one for the first command and one for the second one, leaving the user to match those.
$ sudo cumin -m async 'R:File = /etc/ssl/localcerts/api.svc.codfw.wmnet.chained.crt' 'openssl pkey -pubout -in /etc/ssl/private/api.svc.codfw.wmnet.key | openssl md5' 'openssl x509 -pubkey -in /etc/ssl/localcerts/api.svc.codfw.wmnet.chained.crt -noout | openssl md5'
55 hosts will be targeted:
mw[2120-2147,2200-2223,2251-2253].codfw.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(110) mw[2120-2147,2200-2223,2251-2253].codfw.wmnet
----- OUTPUT -----
(stdin)= c51627f0b52a4dc70d693acdfdf4384a
================
PASS |████████████████████████████████████████████████████████████████████████████████████████████████| 100% (55/55) [00:00<00:00, 89.83hosts/s]
FAIL |                                                                                                         |   0% (0/55) [00:00<?, ?hosts/s]
100.0% (55/55) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Check MySQL semi-sync replication status

Check semi-sync replication status (number of connected clients) on all core mediawiki master databases:

$ sudo cumin 'O:Mariadb::Groups%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\""

Upgrade Debian packages

From time to time it is necessary to roll out new versions of Debian packages. If the version isn't explicitly stated in puppet with ensure then the package won't be normally upgraded.

NB Make sure to test the upgrade on a few selected nodes first.

NB2 Even better, use debdeploy in production to run package upgrades. 

cumin 'HOSTS' 'DEBIAN_FRONTEND=noninteractive apt-get -q -y --assume-no -o \
  DPkg::Options::="--force-confdef" -o DPkg::Options::="--force-confold" \
  install PACKAGE'

Check EtcdConfig for MediaWiki hosts

Check that the part of mediawiki-config that comes from etcd, and is exposed as wmf-config in siteinfo is in sync with etcd checking that the lastModifiedIndex is the last one. The lastModifiedIndex is expected to be different in the two etcd clusters (eqiad vs codfw).

$ sudo cumin 'C:mediawiki::conftool' "curl -sx \$(hostname -f):80 -H'X-Forwarded-Proto: https' 'http://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&format=json&formatversion=2' | jq -r '.query.general[\"wmf-config\"].wmfEtcdLastModifiedIndex'"

To check the actual content of the wmf-config section of the configuration, just remove the .wmfEtcdLastModifiedIndex at the end.

Troubleshooting Production issues

PuppetDB is down

If PuppetDB is not working for some reason (host down, software problems, etc.) Cumin will fail to match hosts based on compound expressions. The Known Hosts and Direct backends will still work using the global grammar syntax with K{} (see Cumin#SSH Known Hosts files backend) and D{} respectively. Alternatively the --backend knownhosts and --backend direct option can also be used respectively. It might make sense in any case to fallback to the secondary PuppetDB host. For a quick fix disable puppet and edit /etc/cumin/config.yaml, in the puppetdb section, amend the: 

   host: puppetdb1001.eqiad.wmnet

to be: 

   host: puppetdb2001.codfw.wmnet

and try to see if cumin is working again. For a permanent fix adjust the hiera variable profile::cumin::master::puppetdb_host.

How to contribute to Cumin

Did you find a bug?

  • If you're unable to find a related issue, open a new one in Phabricator:
    • For WMF Production and WMCS Cloud VPS global installations, use this template.
    • For any Cumin usage outside WMF infrastructure and for WMCS Cloud VPS single project installations, use this template.

Contributing to the code

  • Cumin's development is done on Wikimedia's Gerrit. A developer account is needed to send a pull request.
  • If it's a patch to fix a bug, make sure that a bug was already reported and confirmed, see the above section.
  • if it's a patch for a new feature, make sure to open an issue first and that there is consensus to add this new feature to Cumin.
  • Check the Development page in Cumin's documentation to have a quick overview of its structure.
  • Make sure that all the tests are passing, including the integration tests, if possible.
  • Make sure to reference the existing open issue(s) related to the patch adding a line with Bug: T00001 as the last line of your commit message (replacing the issue ID with the correct one(s)).
  • If you need help, feel free to contact mw:User:RCoccioli_(WMF).

See Also

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Cumin&oldid=460091"