You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

shortcut: mwgrep

Wikimedia binaries: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Legoktm
(→‎ scap pull: formerly sync-common)
imported>Paladox
No edit summary
Line 1: Line 1:
{{Outdated}}{{TOC|align=right}}
{{Outdated}}{{TOC|align=right}}
The following executable '''binaries''' are maintained in version control ([https://git.wikimedia.org/tree/operations%2Fpuppet.git/production/files%2Fmisc%2Fscripts operations/puppet:<code>/files/misc/scripts</code>]) and copied/synced to servers (such as [[tin]]) by puppet into <code>/usr/local/bin</code>.
The following executable '''binaries''' are maintained in version control ([https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/files/misc/scripts operations/puppet:<code>/files/misc/scripts</code>]) and copied/synced to servers (such as [[tin]]) by puppet into <code>/usr/local/bin</code>.


== bin ==
== bin ==
Line 8: Line 8:


=== <code>apache-graceful-all</code> ===
=== <code>apache-graceful-all</code> ===
'''Note:''' this script is in /usr/local/bin/ on tin and [http://git.wikimedia.org/blob/operations%2Fpuppet.git/1e00afe71abf282ee70683a97b349f570dfedd8a/modules%2Fapachesync%2Ffiles%2Fapache-fast-test in puppet].
'''Note:''' this script is in /usr/local/bin/ on tin and [https://phabricator.wikimedia.org/diffusion/OPUP/browse/master/modules/apachesync/files/apache-fast-test;1e00afe71abf282ee70683a97b349f570dfedd8a in puppet].
: ''This script operates on all servers in the following dsh groups: '''apaches'''
: ''This script operates on all servers in the following dsh groups: '''apaches'''
* Runs <code>apache-graceful</code> locally on all apaches.
* Runs <code>apache-graceful</code> locally on all apaches.
Line 21: Line 21:


=== {{anchor|sync-common}}<code>scap pull</code> ===
=== {{anchor|sync-common}}<code>scap pull</code> ===
'''Note:''' this script is from [https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap <code>mediawiki/tools/scap</code>].
'''Note:''' this script is from [https://phabricator.wikimedia.org/diffusion/MWTS/browse/ <code>mediawiki/tools/scap</code>].
: ''Performs the action '''only on the one server''' from which the command is ran''
: ''Performs the action '''only on the one server''' from which the command is ran''
* syncs <code>/srv/mediawiki-staging</code> from a deployment rsync server-> <code>/srv/mediawiki</code> on the local server
* syncs <code>/srv/mediawiki-staging</code> from a deployment rsync server-> <code>/srv/mediawiki</code> on the local server
Line 27: Line 27:


=== {{anchor|sync-dir}}<code>scap sync-dir</code> ===
=== {{anchor|sync-dir}}<code>scap sync-dir</code> ===
'''Note:''' this script is from [https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap <code>mediawiki/tools/scap</code>].
'''Note:''' this script is from [https://phabricator.wikimedia.org/diffusion/MWTS/browse/ <code>mediawiki/tools/scap</code>].
: ''This is an [[#All-script|all-script]]!''
: ''This is an [[#All-script|all-script]]!''
* for a single directory
* for a single directory
Line 34: Line 34:


=== {{anchor|sync-file}}<code>scap sync-file</code> ===
=== {{anchor|sync-file}}<code>scap sync-file</code> ===
'''Note:''' this script is from [https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap <code>mediawiki/tools/scap</code>].
'''Note:''' this script is from [https://phabricator.wikimedia.org/diffusion/MWTS/browse/ <code>mediawiki/tools/scap</code>].
: ''This is an [[#All-script|all-script]]!''
: ''This is an [[#All-script|all-script]]!''
* for a single file
* for a single file
Line 41: Line 41:


=== {{anchor|sync-wikiversions}}<code>scap sync-wikiversions</code>===
=== {{anchor|sync-wikiversions}}<code>scap sync-wikiversions</code>===
'''Note:''' this script is from [https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap <code>mediawiki/tools/scap</code>].
'''Note:''' this script is from [https://phabricator.wikimedia.org/diffusion/MWTS/browse/ <code>mediawiki/tools/scap</code>].
: ''This is an [[#All-script|all-script]]!''
: ''This is an [[#All-script|all-script]]!''
* syncs <code>/srv/mediawiki-staging/wikiversions.{json,cdb}</code> -> <code>/srv/mediawiki</code>
* syncs <code>/srv/mediawiki-staging/wikiversions.{json,cdb}</code> -> <code>/srv/mediawiki</code>
Line 49: Line 49:
[[File:Scap-logo-white-on-black.png|thumb|'scap' logo]]
[[File:Scap-logo-white-on-black.png|thumb|'scap' logo]]
[[File:Scap-diagram.png|thumb|A diagram of wikimedia 'scap' deployment tool]]
[[File:Scap-diagram.png|thumb|A diagram of wikimedia 'scap' deployment tool]]
'''Note:''' this script is from <code>[https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap mediawiki/tools/scap]</code>. Generated documentation is [https://doc.wikimedia.org/mw-tools-scap/ published on doc.wikimedia.org]
'''Note:''' this script is from <code>[https://phabricator.wikimedia.org/diffusion/MWTS/browse/ mediawiki/tools/scap]</code>. Generated documentation is [https://doc.wikimedia.org/mw-tools-scap/ published on doc.wikimedia.org]
: ''This is an [[#All-script|all-script]]!''
: ''This is an [[#All-script|all-script]]!''
: ''"sync-common-all-php"''
: ''"sync-common-all-php"''

Revision as of 16:32, 1 July 2016

The following executable binaries are maintained in version control (operations/puppet:/files/misc/scripts) and copied/synced to servers (such as tin) by puppet into /usr/local/bin.

bin

apache-graceful

Note: this script is NOT in puppet.

Performs the action only on the one server from which the command is ran

apache-graceful-all

Note: this script is in /usr/local/bin/ on tin and in puppet.

This script operates on all servers in the following dsh groups: apaches
  • Runs apache-graceful locally on all apaches.

fatalmonitor

  • Runs a watch command on the apache logs with various exclusion and aggregation filters to monitor current php notices, warnings and errors on the cluster.

sync-apache

This script operates on all servers in the following dsh groups: apaches, image_scalers, snapshot, searchidx
Note: The conf files are outside of /a/common, and as such are not synced by the sync-common-like scripts.
  • syncs /home/wikipedia/conf/httpd to /apache/conf

scap pull

Note: this script is from mediawiki/tools/scap.

Performs the action only on the one server from which the command is ran
  • syncs /srv/mediawiki-staging from a deployment rsync server-> /srv/mediawiki on the local server
  • This action was formerly run as sync-common

scap sync-dir

Note: this script is from mediawiki/tools/scap.

This is an all-script!
  • for a single directory
  • checks PHP syntax
  • syncs /srv/mediawiki-staging/(some dir) -> /srv/mediawiki

scap sync-file

Note: this script is from mediawiki/tools/scap.

This is an all-script!
  • for a single file
  • checks PHP syntax
  • syncs /srv/mediawiki-staging/(some file) -> /srv/mediawiki

scap sync-wikiversions

Note: this script is from mediawiki/tools/scap.

This is an all-script!
  • syncs /srv/mediawiki-staging/wikiversions.{json,cdb} -> /srv/mediawiki

scap sync

'scap' logo
A diagram of wikimedia 'scap' deployment tool

Note: this script is from mediawiki/tools/scap. Generated documentation is published on doc.wikimedia.org

This is an all-script!
"sync-common-all-php"
  • lints PHP files in ./wmf-config, ./multiversion
  • Sync deploy directory (/srv/mediawiki-staging) on localhost with staging area (/srv/mediawiki)
  • for all mediawiki versions currently deployed (usually 2) it:
    • rebuilds localization caches of core and extensions
    • creates json files from localization caches for better rsync performance
  • runs scap pull on co-master servers to update /srv/mediawiki-staging
  • runs scap pull on rsync fanout servers to update /srv/mediawiki
  • runs scap pull on "all" servers to update /srv/mediawiki from nearest rsync slave
  • rebuilds localization caches on "all" servers from json files that were synced by scap pull
  • compiles wikiversions.json to wikiversions.cdb on localhost
  • runs scap sync-wikiversions
bd808@silver:~$ scap say
 ------------------------------------------------
/                                                \
|     S.C.A.P.: someone can always pontificate    |
\                                                /
 ------------------------------------------------
    \
     \
      \
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

mwgrep

Search for CSS and JavaScript code fragments in MediaWiki, User and Module namespaces pages on all Wikimedia projects (powered by Elasticsearch).

It's located in operations/puppet/modules/scap/files/mwgrep

All-script

An all-script is a script that operates on mediawiki-installation boxes via ssh. These scripts performs the action on all servers.

The specific servers include:

Examples

sync-apache example

  1. On tin:
  2. (optionally) run:
    sync-apache-simulated
    This just has an "-n" added to the rsync commands. (--dry-run | show what would have been transferred)
  3. If that looks fine, actually sync:
    sync-apache
  4. Connect to a single random "srv" server, confirm the expected change has arrived here, and restart Apache to check your change did not break things.
    1. (how to restart that apache?)
  5. If that also looks fine, restart all Apaches:
    apache-graceful-all

After this you might want to purge a URL from Squid cache. Also see Squids#Purge a given external (non WMF) url.

In most cases this is enough. However if the change involves more than a simple change to redirects, depooling/ repooling may be necessary. Make sure you find out whether this is needed ahead of time.

See also