You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Difference between revisions of "Scap3"

From Wikitech-static
Jump to navigation Jump to search
imported>Effie Mouzeli
imported>Thcipriani
(Redirected page to Scap#Other software deployments)
 
Line 1: Line 1:
For information about deploying new software with Scap3 see the [[Scap3/Deployment|Scap3 Deployment Guide]].
#REDIRECT [[Scap#Other_software_deployments]]
 
For information about migrating existing services from Trebuchet to Scap3 see the [[Scap3/Migration_Guide|Scap3 Migration Guide]]
 
= Scap3 Documentation =
 
This is a collection of some of the documentation contained in the commit messages of merged commits that were part of the 'scap3' sprint. They are collected here for ease of access and search-ability.
 
The primary documentation for scap can be found in the form of auto-generated html docs, see [https://doc.wikimedia.org/mw-tools-scap/index.html scap documentation index].
 
== Basic Use ==
 
The first step to deploying new code from the deployment_host is to use <code>git</code> to bring the repository on the deployment host into the state that you want deployed to your targets.
 
Once the repository state is correct, use the deploy command to release the finished code use: <code>scap deploy -v</code>
 
<syntaxhighlight lang=shell-session>
deployer@tin:/srv/deployment/foo/deploy$ scap deploy
20:46:12 Started deploy_foo/deploy
Entering 'foo'
20:46:12
== DEFAULT ==
:* scap-target-07
:* scap-target-08
:* scap-target-09
:* scap-target-04
:* scap-target-05
:* scap-target-06
:* scap-target-10
:* scap-target-01
:* scap-target-02
:* scap-target-03
deploy_foo/deploy_config_deploy: 100% (ok: 10; fail: 0; left: 0)
deploy_foo/deploy_fetch: 100% (ok: 10; fail: 0; left: 0)
deploy_foo/deploy_promote: 100% (ok: 10; fail: 0; left: 0)
20:46:42 Finished deploy_foo/deploy (duration: 00m 29s)
</syntaxhighlight>
 
== Service Checks ==
 
=== NRPE checks ===
   
<code>nrpe</code> check type for reusing existing Icinga/NRPE checks in deployments.
   
By default, checks are loaded and registered from all NRPE definitions in <code>/etc/nagios/nrpe.d</code> and can be referenced in <code>checks.yaml</code> using <code>type: nrpe</code>.
 
<pre>checks:
  service_endpoints:
    type: nrpe
    command: check_service_endpoints
    stage: promote
</pre>
 
== [https://gerrit.wikimedia.org/r/#/c/240292/ Config Deployments] ==
 
=== What happens on the deployment server ===
* Looks for environment-specific <code>./scap/config-files.yaml</code> with target files in the format:
<pre>
/path/to/target:
  template: env-specific-template.yaml.j2
  remote_vars: /optional/remote/variable/file.yaml
</pre>
* Looks for environment-specific <code>./scap/vars.yaml</code> that includes variables used to render the template. These variables will be overridden by any conflicting variables in the file specified by <code>remote_vars</code>
* Variables from any environment-specific <code>vars.yaml</code> file are combined with variables from the root <code>vars.yaml</code> file.
* A json file is created at <code>[repo]/.git/config-files/[revision].json</code> that contains the final path to any environment-specific templates as well as a final list of combined variables.
 
=== What happens on targets ===
 
* Download the file from <code>[deployment-server]/[repo]/.git/config-files/[revision].json</code>
* Loop through the config files, and render each template using the variables from the downloaded json file and the variables from the (now) local <code>remote_vars</code> file
* Links rendered file (in <code>[repo]/.git/config-files/[path]</code>) to final location
 
== Grouping and Filtering deployment targets ==
 
=== [https://phabricator.wikimedia.org/rMSCAf3787852a56fcbf87784de9b3b3585138fe003c0 Filter deploy hosts with --limit-hosts] ===
 
<code>--limit-hosts</code> or <code>-l</code> accepts a pattern in one of the following formats:
* <code>~[regex]</code> - if the pattern starts with ~ it is interpreted as a regular expression, this is not combined with any other option
* <code>![pattern]</code> - can be combined with any other pattern to negate that pattern. May be combined with other pattern matching options.
* <code>host[01:10]</code> - range matching, works for ascii chars and numbers, including numbers with a leading 0, may be combined with other pattern-matching options.
* <code>host*</code> - Matches 0 or more characters in the set A-z, '_', '.', or '-'. May be combined with other pattern matching options. This pattern is applied to the <code>dsh_targets</code> file to return a sub-set of hosts to use as a deployment target.
 
=== [https://phabricator.wikimedia.org/D16 Deployment groups] ===
 
This feature was introduced in differential revision [https://phabricator.wikimedia.org/D16 D16]
 
In addition to the <code>dsh_targets</code> config variable, scap looks for multiple <code>''[anything]''_dsh_targets</code> config variables. This enables <code>canary_dsh_targets</code>.
 
All additional deployment groups will be executed before the primary deployment group (defined by the <code>dsh_targets</code> variable).
 
Additionally, checks now can be scoped to a specific deployment group using:
 
<pre>check_name:
    stage: promote
    group: dsh-group-name
    command: touch /tmp/hi-there
</pre>
The group name is optional in a check. If not group name is specified, check runs for all deploy groups.
 
== Structured Logging ==
 
=== Overview ===
 
This feature was introduced in differential revision [https://phabricator.wikimedia.org/D18 D18]
 
The main deploy application now sends all structured log output to a
file under <code>scap/log/{git-tag}.log</code> which the new deploy-log utility
can tail and filter using a given free-form expression. By default the
latter utility will periodically scan the <code>scap/log</code> directory for new
files and immediately begin tailing them. It can also be given an
explicit log file to parse via the <code>--file</code> option or the latest log
file by using <code>--latest</code>; in this case, it will simply filter the entire
file for matching records and exit.
 
=== Examples ===
 
==== Tail behavior ====
 
* Run <code>deploy-log {expr}</code>
* Run <code>deploy</code> in a separate terminal
* Verify that **deploy-log** in the first terminal starts reading the new log file. It should say -- Opening log file: <code>{file}</code>.
* Verify that only log messages matching the given expression are output.
 
==== Latest log file behavior ====
 
* Run <code>deploy</code>.
* Run <code>deploy-log -l {expr}</code>
* Verify that only log messages from the latest log file matching the given expression are output.
 
==== Single log file behavior ====
 
* Run <code>deploy</code> a couple of times.
* Run <code>deploy-log -f {log-file} {expr}</code>
* Verify that only log messages from the given log file matching the given expression are output.
 
== Production Upgrade ==
 
=== Building ===
To prepare a new release of the Debian package for Scap, the '''Release Engineering Team''' needs to follow the steps in [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/RELEASE.md RELEASE.md] (in the Scap git repository), and then the '''SRE team''' needs to build and deploy the package, which are listed below.
 
The Debian package for scap can be built with <code>git-buildpackage</code>. More specifically: for production the standard procedure is to have a Debian source package built on the <tt>package_builder</tt> machine.
 
  git clone <nowiki>https://gerrit.wikimedia.org/r/mediawiki/tools/scap</nowiki>
  cd scap
  V=3.17.1 # Change value as appropriate
  git checkout -b $V $V
  WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=buster --git-debian-branch=$V
 
=== Uploading to apt repos ===
 
The resulting package will be in <tt>/var/cache/pbuilder/result/stretch-amd64/</tt> and needs to be uploaded to the apt repo (e.g. from <tt>apt1001.wikimedia.org</tt>)
 
  export DIST="buster"
  rsync -vaz deneb.codfw.wmnet::pbuilder-result/$DIST-amd64/*scap* deb/
  sudo -i reprepro --ignore=wrongdistribution include $DIST-wikimedia $(pwd)/deb/scap_<VERSION>_amd64.changes
  # scap is compatible as-is with jessie, stretch and buster, just copy the package there
  sudo -i reprepro copy jessie-wikimedia $DIST-wikimedia scap
  sudo -i reprepro copy stretch-wikimedia $DIST-wikimedia scap
 
If you get errors about a missing GPG key and therefore not exporting indices you need to import the right key like described on [[Reprepro#Adding_a_new_external_repository]].
 
If signing still fails after you imported the key you need to ensure it looks for them in the right home dir, see [[Reprepro#If_signing_fails]].
 
If this already happened and you can't get it to export indices by repeating the "include" command you can work around it by copying it back from another distro (reprepro copy stretch-wikimedia buster-wikimedia scap).
 
Not having exported indices will manifest as "reprepro ls shows the right version but it is not found on a client even after running apt-get update".
 
=== Roll out to production ===
You will then need to use [[Software_deployment#Deploying_a_software_update_fleet-wide_using_debdeploy_and_debmonitor|Debdeploy]] to deploy the new package in production.
 
* Start by deploying to <code>mw-api-canary, mw-canary, mw-jobrunner-canary</code> (cumin aliases), then login to '''<code>mwdebug*</code>''' servers and check there if a '''<code>scap pull</code>''' still works.
* Later or the next day you can rollout to <code>all</code>.
=== Resources ===
There are Debian [https://www.debian.org/doc/debian-policy/ch-R:Package = scapcontrolfields.html#s-f-Standards-Version Versioning guidelines]
 
[[Category:Deployment]]

Latest revision as of 20:33, 14 September 2021