You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Wikimedia Cloud Services team/EnhancementProposals/Operational Automation: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>BryanDavis
(whitespace cleanup)
imported>Majavah
(fix headings (level 1 is used for page title and should not be used for anything else without a good reason), use syntax highlight tags for code blocks)
Line 15: Line 15:


=== Collaboration ===
=== Collaboration ===
The main advantage of choosing spicerack is collaboration with the rest of the SRE teams. This comes with both the duty and privilege of becoming co-maintainers for spicerack and related projects, allowing us to have a say in the direction of the project and the use cases that will be supported. With the duty of driving, reviewing and maintaining the projects for all the users (including other SRE teams).
The main advantage of choosing spicerack is collaboration with the rest of the SRE teams. <s>This comes with both the duty and privilege of becoming co-maintainers for spicerack and related projects, allowing us to have a say in the direction of the project and the use cases that will be supported. With the duty of driving, reviewing and maintaining the projects for all the users (including other SRE teams).</s>


=== Structure ===
=== Structure ===
Line 28: Line 28:
[[Python/Wmflib|Wmflib]] is a bundle of generic commonly used functions related to the wikimedia foundation, has some helpful decorators (ex. retry) and similar tools.
[[Python/Wmflib|Wmflib]] is a bundle of generic commonly used functions related to the wikimedia foundation, has some helpful decorators (ex. retry) and similar tools.


This library should be used for generic functions that are not bound to the spicerack library and can be reused in other non-spicerack related projects easily.
This library should be used for generic functions that are not bound to the spicerack library and can be reused in other non-spicerack wikimedia related projects.


==== Spicerack ====
==== Spicerack ====
[[Spicerack]] is the core library, contains more wikimedia specific libraries and cli (cookbook) to interact with different services (ex. toolforge.etcd) and is meant to be used to store the core logic for any interaction with the services.
[[Spicerack]] is the core library, contains more wikimedia specific libraries and a cli (cookbook) to interact with different services (ex. toolforge.etcd) and is meant to be used to store the core logic for any interaction with the services.


Here we will have to add, specially at the beginning, some libraries to interact with our services, here will also be where more of the reusage of code and collaboration will happen. We should keep always in mind things here that can be used by other group around the foundation. Code in this library will be considerably tested, and no merges should happen without review.
Here we will have to add, specially at the beginning, some libraries to interact with our services, here will also be where more of the re-usage of code and collaboration will happen. We should keep always in mind things here that can be used by other group around the foundation. Code in this library will be considerably tested, and no merges should happen without review.


==== Cookbooks ====
==== Cookbooks ====
The [[Spicerack/Cookbooks]] repo contains the main recipes to execute, the only logic should be orchestration, and any service management related code should be eventually moved to the above Spicerack library.
The [[Spicerack/Cookbooks]] repo contains the main recipes to execute, the only logic should be orchestration, and any service management related code should be eventually moved to the above Spicerack library.


This repository of cookbooks will be shared with the rest of the SRE group, but our specific cookbooks will go under the `cookbooks/wmcs` directory.
This repository of cookbooks will be shared with the rest of the SRE group, but our specific cookbooks will go under the <code>cookbooks/wmcs</code> directory.
Any helper library should go under `cookbooks/wmcs/__init__.py` and we should periodically consider moving as much of the code from there to Spicerack.
Any helper library should go under <code>cookbooks/wmcs/__init__.py</code> and we should periodically consider moving as much of the code from there to Spicerack.


=== Execution of the cookbooks ===
=== Execution of the cookbooks ===
Line 49: Line 49:
To run locally the cookbooks, you will need to create a virtualenv and install the dependencies, for example:
To run locally the cookbooks, you will need to create a virtualenv and install the dependencies, for example:


    mkvirtualenv spicerack
<syntaxhighlight lang=shell-session>
    pip install wikimedia-spicerack
$ mkvirtualenv spicerack
 
$ pip install wikimedia-spicerack
NOTE: as of writing this, we are pending a release of spicerack, so for this to work you might have to clone spicerack locally and install it, see (https://gerrit.wikimedia.org/r/admin/repos/operations/software/spicerack):
</syntaxhighlight>
 
    git clone "https://dcaro@gerrit.wikimedia.org/r/a/operations/software/spicerack"
    cd spicerack
    pip uninstall wikimedia-spicerack
    pip install -e .


Then clone the cookbooks repo (see https://gerrit.wikimedia.org/r/admin/repos/operations/cookbooks):
Then clone the cookbooks repo (see https://gerrit.wikimedia.org/r/admin/repos/operations/cookbooks):


    git clone "https://dcaro@gerrit.wikimedia.org/r/a/operations/cookbooks"
<syntaxhighlight lang=shell-session>
$ git clone "https://$USER@gerrit.wikimedia.org/r/a/operations/cookbooks"
</syntaxhighlight>


Then create the relevant config files, one for cumin, wherever you prefer, I recommend `~/.config/spicerack/cumin_config.yaml`, with the contents:
Then create the relevant config files, one for cumin, wherever you prefer, I recommend <code>~/.config/spicerack/cumin_config.yaml</code>, with the contents:


    ---
<syntaxhighlight lang=yaml>
    transport: clustershell
---
    log_file: cumin.log  # feel free to change this to another path of your choosing
transport: clustershell
    default_backend: direct
log_file: cumin.log  # feel free to change this to another path of your choosing
    clustershell:
default_backend: direct
        ssh_options:
environment: {}
            # needed for vms that repeat a name
clustershell:
            - |
    ssh_options:
                -o StrictHostKeyChecking=no
        # needed for vms that repeat a name
                -o "UserKnownHostsFile=/dev/null"
        - |
                -o "LogLevel=ERROR"
            -o StrictHostKeyChecking=no
            -o "UserKnownHostsFile=/dev/null"
            -o "LogLevel=ERROR"
</syntaxhighlight>


And another for spicerack itself (the cookbook cli), I'll use `~/.config/spicerack/cookbook_config.yaml`
And another for spicerack itself (the cookbook cli), I'll use <code>~/.config/spicerack/cookbook_config.yaml</code>


    ---
<syntaxhighlight lang=yaml>
    # adapt to wherever you cloned the repo
---
    cookbooks_base_dir: ~/Work/wikimedia/operations/cookbooks
# adapt to wherever you cloned the repo
    instance_params:
cookbooks_base_dir: ~/Work/wikimedia/operations/cookbooks
        cumin_config: ~/.config/spicerack/cumin-config.yaml
instance_params:
    cumin_config: ~/.config/spicerack/cumin-config.yaml
</syntaxhighlight>




With those config files, now you are able to run the client, from the root of the operations/cookbooks repository, you can list all the cookbooks:
With those config files, now you are able to run the client, from the root of the <code>operations/cookbooks</code> repository, you can list all the cookbooks:


    cookbook -c ~/.config/spicerack/cookbook_config.yaml --list
<syntaxhighlight lang=shell-session>
$ cookbook -c ~/.config/spicerack/cookbook_config.yaml --list
</syntaxhighlight>


NOTE: This is a very premature proposal, this workflow will be improved, feel free to start any questions/discussions in the talks page or ping me directly ([[User:David Caro|David Caro]] ([[User talk:David Caro|talk]]) 17:21, 5 February 2021 (UTC)).
NOTE: This is a very premature proposal, this workflow will be improved, feel free to start any questions/discussions in the talks page or ping me directly ([[User:David Caro|David Caro]] ([[User talk:David Caro|talk]]) 17:21, 5 February 2021 (UTC)).

Revision as of 13:20, 20 April 2021

We currently use Puppet to automate most of our tasks, but it has it's limitations. We still need a tool to automate, collect and review all our operational procedures. Some examples of such procedures are:

  • Adding a new member to a toolforge instance etcd cluster.
  • Bootstrapping a new toolforge instance.
  • Searching for the host where a backup is kept.
  • Provisioning a new cloudvirt node.
  • Re-image all/a set of the cloudvirt nodes.
  • Manage non-automated upgrades.
  • Take down a cloudvirt node for maintenance.

Problem statement

All these tasks still require manual operations, following a runbook whenever available, that easily get outdated, are prone to human error and require considerable attention to execute.

Proposal

After reviewing several automation tools (spicerack, saltstack, puppet, ...), and doing a quick POC (see gerrit:647735 and gerrit:658637) for the two more relevant (summary of the experience here https://etherpad.wikimedia.org/p/ansible_vs_spicerack), I've decided to propose spicerack as the de-facto tool for WMCS operational tasks automation.

Collaboration

The main advantage of choosing spicerack is collaboration with the rest of the SRE teams. This comes with both the duty and privilege of becoming co-maintainers for spicerack and related projects, allowing us to have a say in the direction of the project and the use cases that will be supported. With the duty of driving, reviewing and maintaining the projects for all the users (including other SRE teams).

Structure

The Spicerack ecosystem is split in several projects:

Cumin

Cumin is the lowermost layer, built on top of ClusterShell takes care of translating host expressions to hosts, running commands in them (using whatever strategy is selected) and returning the results.

This library should be pretty stable and require little to no changes.

Wmflib

Wmflib is a bundle of generic commonly used functions related to the wikimedia foundation, has some helpful decorators (ex. retry) and similar tools.

This library should be used for generic functions that are not bound to the spicerack library and can be reused in other non-spicerack wikimedia related projects.

Spicerack

Spicerack is the core library, contains more wikimedia specific libraries and a cli (cookbook) to interact with different services (ex. toolforge.etcd) and is meant to be used to store the core logic for any interaction with the services.

Here we will have to add, specially at the beginning, some libraries to interact with our services, here will also be where more of the re-usage of code and collaboration will happen. We should keep always in mind things here that can be used by other group around the foundation. Code in this library will be considerably tested, and no merges should happen without review.

Cookbooks

The Spicerack/Cookbooks repo contains the main recipes to execute, the only logic should be orchestration, and any service management related code should be eventually moved to the above Spicerack library.

This repository of cookbooks will be shared with the rest of the SRE group, but our specific cookbooks will go under the cookbooks/wmcs directory. Any helper library should go under cookbooks/wmcs/__init__.py and we should periodically consider moving as much of the code from there to Spicerack.

Execution of the cookbooks

As of now, this cookbooks can be run locally, I'm actively considering how to provide a host/vm/... with a spicerack + cumin setup for easy usage and running long cookbooks, but as of right now, we can start locally.

If your cookbooks are only accessing bare metal machines, you can already run them on the cumin hosts for the wiki operations (ex. cumin1001.eqiad.wmnet), but those hosts have no access to the VMs as of writing this.

Local setup

To run locally the cookbooks, you will need to create a virtualenv and install the dependencies, for example:

$ mkvirtualenv spicerack
$ pip install wikimedia-spicerack

Then clone the cookbooks repo (see https://gerrit.wikimedia.org/r/admin/repos/operations/cookbooks):

$ git clone "https://$USER@gerrit.wikimedia.org/r/a/operations/cookbooks"

Then create the relevant config files, one for cumin, wherever you prefer, I recommend ~/.config/spicerack/cumin_config.yaml, with the contents:

---
transport: clustershell
log_file: cumin.log  # feel free to change this to another path of your choosing
default_backend: direct
environment: {}
clustershell:
    ssh_options:
        # needed for vms that repeat a name
        - |
            -o StrictHostKeyChecking=no
            -o "UserKnownHostsFile=/dev/null"
            -o "LogLevel=ERROR"

And another for spicerack itself (the cookbook cli), I'll use ~/.config/spicerack/cookbook_config.yaml

---
# adapt to wherever you cloned the repo
cookbooks_base_dir: ~/Work/wikimedia/operations/cookbooks
instance_params:
    cumin_config: ~/.config/spicerack/cumin-config.yaml


With those config files, now you are able to run the client, from the root of the operations/cookbooks repository, you can list all the cookbooks:

$ cookbook -c ~/.config/spicerack/cookbook_config.yaml --list

NOTE: This is a very premature proposal, this workflow will be improved, feel free to start any questions/discussions in the talks page or ping me directly (David Caro (talk) 17:21, 5 February 2021 (UTC)).