You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Cumin: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Volans
m (Add disclaimer.)
imported>Volans
(Add some paragraphs of general documentation)
Line 1: Line 1:
An automation and orchestration framework
==== Automation and orchestration framework written in Python ====


Given that Cumin's grammar will probably be subject to changes in the near future to allow to cover more usages and simply the most common usages, the full documentation here will be added later as part of https://phabricator.wikimedia.org/T158964
== Features ==
For a general description of Cumin's features, see https://github.com/wikimedia/cumin/blob/master/README.md


See https://github.com/wikimedia/cumin/blob/master/README.md for now.
The '''TL;DR''' quick summary of Cumin features, relevant to the usage inside WMF are:
* '''Select''' target hosts by name and querying PuppetDB for any included Puppet Resource or Puppet Fact.
* '''Execute''' any number of arbitrary commands via SSH on the selected target hosts in an orchestrated way (see below) grouping the hosts the have the same output


== Examples of usage on WMF ==
== Host selection ==
'''DISCLAIMER:''' the grammar used by Cumin to query its backends for the hosts selection will be modified in the near future to improve the capabilities of host selection. In particular it will be expanded to encapsulate the current grammar into composable blocks to allow more complex and powerful host selection queries. This documentation will be updated accordingly.
* Match hosts by name with a simple globbing:
** <code>wdqs2*</code>  matches all the hosts with hostname starting with <code>wdqs2</code> hence all the Wikidata Query Service hosts in codfw. <code>wdqs2*.codfw.wmnet</code> is a more formal way to specify it.
** <code>wdqs2* or pc2*</code>  matches all the above plus the codfw's Parser Cache hosts.
* Match hosts using the [http://clustershell.readthedocs.io/en/latest/api/NodeSet.html#ClusterShell.NodeSet.NodeSet ClusterShell NodeSet] syntax:
** <code>db[2016-2019,2023,2028-2029,2033].codfw.wmnet</code> define a specific list of hosts
* Puppet Fact selection:
** <code>F:memorysize_mb ~ "^[2-3][0-9][0-9][0-9][0-9]"</code> selects all the hosts that have beween 20000MB and 39999MB of RAM
** <code>F:lsbdistid = Ubuntu and analytics*</code> selects all the hosts with hostname that starts with <code>analytics</code> that have Ubuntu as OS
* Puppet Resource selection:
** <code>R:File = /etc/ssl/localcerts/api.svc.eqiad.wmnet.chained.crt</code> selects all the hosts in which Puppet manages this specific file resource
** <code>R:Class = Mediawiki::Nutcracker and *.eqiad.wmnet</code> selects all the hosts that have the Puppet Class <code>Mediawiki::Nutcracker</code> applied and the hostname ending in <code>.eqiad.wmnet</code>, that is a quick hack to select a single datacenter if there are not hosts <code>.wikimedia.org</code> involved until we'll expose <code>$::site</code> and other global variables to PuppetDB.
* Special all hosts matching: <code>*</code> '''!!!ATTENTION: use extreme caution with this selector!!!'''
 
== Command execution ==
TODO
 
== WMF installation ==
 
=== Production infrastructure ===
In the WMF production infrastructure, Cumin masters are installed via Puppet's <code>Role::Cumin::Master</code> role, that is currently included in the <code>Role::Cluster::Management</code> role. Cumin can be executed in any of those hosts and requires '''sudo''' privileges or being root. Cumin can access any production host that includes the <code>Profile::Cumin::Target</code> profile as root (all production hosts as of now), hence is a very powerful but also a potentially very dangerous tool, '''be very careful''' while using it. The current Cumin's masters from where it can be executed are:
{| class="wikitable"
!Cumin master hosts
|-
|<code>neodymium.eqiad.wmnet</code>
|-
|<code>sarin.codfw.wmnet</code>
|}
The default Cumin backend is configured to be PuppetDB and the default transport ClusterShell (SSH). The capability of Cumin to query PuppetDB as a backend allow to select hosts in a very powerful and precise way, querying for any Puppet resource or fact. Mixed query for resources and facts are currently not supported, but will be addressed by the grammar improvements described above.
 
== Examples of usage in the WMF infrastructure ==


* Check semi-sync replication status (number of connected clients) on all core mediawiki master databases:
* Check semi-sync replication status (number of connected clients) on all core mediawiki master databases:
  root@neodymium:~$ cumin 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\""
  sudo cumin 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\""

Revision as of 22:31, 29 March 2017

Automation and orchestration framework written in Python

Features

For a general description of Cumin's features, see https://github.com/wikimedia/cumin/blob/master/README.md

The TL;DR quick summary of Cumin features, relevant to the usage inside WMF are:

  • Select target hosts by name and querying PuppetDB for any included Puppet Resource or Puppet Fact.
  • Execute any number of arbitrary commands via SSH on the selected target hosts in an orchestrated way (see below) grouping the hosts the have the same output

Host selection

DISCLAIMER: the grammar used by Cumin to query its backends for the hosts selection will be modified in the near future to improve the capabilities of host selection. In particular it will be expanded to encapsulate the current grammar into composable blocks to allow more complex and powerful host selection queries. This documentation will be updated accordingly.

  • Match hosts by name with a simple globbing:
    • wdqs2* matches all the hosts with hostname starting with wdqs2 hence all the Wikidata Query Service hosts in codfw. wdqs2*.codfw.wmnet is a more formal way to specify it.
    • wdqs2* or pc2* matches all the above plus the codfw's Parser Cache hosts.
  • Match hosts using the ClusterShell NodeSet syntax:
    • db[2016-2019,2023,2028-2029,2033].codfw.wmnet define a specific list of hosts
  • Puppet Fact selection:
    • F:memorysize_mb ~ "^[2-3][0-9][0-9][0-9][0-9]" selects all the hosts that have beween 20000MB and 39999MB of RAM
    • F:lsbdistid = Ubuntu and analytics* selects all the hosts with hostname that starts with analytics that have Ubuntu as OS
  • Puppet Resource selection:
    • R:File = /etc/ssl/localcerts/api.svc.eqiad.wmnet.chained.crt selects all the hosts in which Puppet manages this specific file resource
    • R:Class = Mediawiki::Nutcracker and *.eqiad.wmnet selects all the hosts that have the Puppet Class Mediawiki::Nutcracker applied and the hostname ending in .eqiad.wmnet, that is a quick hack to select a single datacenter if there are not hosts .wikimedia.org involved until we'll expose $::site and other global variables to PuppetDB.
  • Special all hosts matching: * !!!ATTENTION: use extreme caution with this selector!!!

Command execution

TODO

WMF installation

Production infrastructure

In the WMF production infrastructure, Cumin masters are installed via Puppet's Role::Cumin::Master role, that is currently included in the Role::Cluster::Management role. Cumin can be executed in any of those hosts and requires sudo privileges or being root. Cumin can access any production host that includes the Profile::Cumin::Target profile as root (all production hosts as of now), hence is a very powerful but also a potentially very dangerous tool, be very careful while using it. The current Cumin's masters from where it can be executed are:

Cumin master hosts
neodymium.eqiad.wmnet
sarin.codfw.wmnet

The default Cumin backend is configured to be PuppetDB and the default transport ClusterShell (SSH). The capability of Cumin to query PuppetDB as a backend allow to select hosts in a very powerful and precise way, querying for any Puppet resource or fact. Mixed query for resources and facts are currently not supported, but will be addressed by the grammar improvements described above.

Examples of usage in the WMF infrastructure

  • Check semi-sync replication status (number of connected clients) on all core mediawiki master databases:
sudo cumin 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\""