You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

shortcut: sql

Wikimedia binaries: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>BryanDavis
(Rename scap sub-commands; add {{anchor}} for old labels to preserve deep links; add co-master sync for #scap_sync)
imported>Krinkle
(27 intermediate revisions by 17 users not shown)
Line 1: Line 1:
{{Outdated}}{{TOC|align=right}}
{{Outdated}}{{TOC|align=right}}
The following executable '''binaries''' are maintained in version control ([https://git.wikimedia.org/tree/operations%2Fpuppet.git/production/files%2Fmisc%2Fscripts operations/puppet:<code>/files/misc/scripts</code>]) and copied/synced to servers (such as [[tin]]) by puppet into <code>/usr/local/bin</code>.


== bin ==
The following executable '''binaries''' are maintained in version control ([[gerrit:plugins/gitiles/operations/puppet/+/production/modules/scap/files|operations/puppet:<code>/modules/scap/files</code>]] and others) and provisioned by Puppet into <code>/usr/local/bin</code>.
=== <code>apache-graceful</code> ===
'''Note:''' this script is NOT in puppet.
: ''Performs the action '''only on the one server''' from which the command is ran''


=== <code>apache-graceful-all</code> ===
== Deployment host ==
'''Note:''' this script is in /usr/local/bin/ on tin and [http://git.wikimedia.org/blob/operations%2Fpuppet.git/1e00afe71abf282ee70683a97b349f570dfedd8a/modules%2Fapachesync%2Ffiles%2Fapache-fast-test in puppet].
These commands on the deployment host ([[deployment.eqiad.wmnet]]).
: ''This script operates on all servers in the following dsh groups: '''apaches'''
 
* Runs <code>apache-graceful</code> locally on all apaches.
=== <code>httpbb</code> ===
View source in [[git:operations/software/httpbb|gerrit]].
 
== mwlog-server ==
These commands must be run from [[mwlog1002]].
 
=== <code>logspam-watch</code> ===
Periodically refreshes and offers a filtering interface to <code>logspam</code>, which in turn aggregates various PHP error and exception logs.


=== <code>fatalmonitor</code> ===
=== <code>fatalmonitor</code> ===
* Runs a [[wikipedia:watch (Unix)|<code>watch</code>]] command on the [[Logs#fenari:/home/wikipedia/syslog|apache logs]] with various exclusion and aggregation filters to monitor current php notices, warnings and errors on the cluster.
No longer exists - see <code>logspam-watch</code> instead.
 
== Maintenance host ==
These commands must be run on a maintenance host (e.g. [[mwmaint1002]]).
 
Consider using [[Screen|'''screen''']] or [[Tmux|'''tmux''']] before starting any long-running maintenance scripts.
 
===<code>mwscript</code>===
Maintenance script runner. This wrapper is '''mandatory''' for all maintenance scripts in production. It takes care of finding where the appropriate MediaWiki installation  on the server for the selected dbname (per [[Multiversion]]), setting environment variables, sudo-ing as the correct shell user, and more. See [[Heterogeneous deployment]] for more about Multiversion.
 
Script names must be relative to a MediaWiki install path, for example:
mwscript maintenance/doExample.php --wiki <nameofwiki> [args]
or
mwscript extensions/Example/doExample.php --wiki <nameofwiki> [args]
 
===<code>mwscriptwikiset</code>===
Sequentially run a maintenance script for all wikis in a dblist-file (in wmf-config/dblists). For example:
mwscriptwikiset maintenance/doExample.php group0.dblist [args]
This only supports expanded or plain dblist files. It does not support dblist files containing a computable expression.


=== <code>sync-apache</code> ===
It effectively runs <code>mwscript maintenance/doExample.php <nameofwiki> [args]</code>.
: ''This script operates on all servers in the following dsh groups: '''apaches, image_scalers, snapshot, searchidx'''
: '''Note:''' The conf files are outside of <code>/a/''common''</code>, and as such are '''not''' synced by the <code>sync-common</code>-like scripts.
* syncs <code>/home/wikipedia/conf/httpd</code> to <code>/apache/conf</code>


=== {{anchor|sync-common}}<code>scap pull</code> ===
=== <code>foreachwikiindblist</code> ===
'''Note:''' this script is from [https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap <code>mediawiki/tools/scap</code>].
Sequentially run a maintenance script for all wikis in a dblist expression.
: ''Performs the action '''only on the one server''' from which the command is ran''
* syncs <code>/srv/mediawiki-staging</code> from a deployment rsync server-> <code>/srv/mediawiki</code> on the local server


=== {{anchor|sync-dir}}<code>scap sync-dir</code> ===
Similar to <code>mwscriptwikiset</code>, except that it supports dblist expressions, and invokes mwscript with <code>--wiki</code> instead of position arguments.
'''Note:''' this script is from [https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap <code>mediawiki/tools/scap</code>].
: ''This is an [[#All-script|all-script]]!''
* for a single directory
* checks PHP syntax
* syncs <code>/srv/mediawiki-staging/(some dir)</code> -> <code>/srv/mediawiki</code>


=== {{anchor|sync-file}}<code>scap sync-file</code> ===
For example:
'''Note:''' this script is from [https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap <code>mediawiki/tools/scap</code>].
foreachwikiindblist <dblist or expression> maintenance/doExample.php [args]
: ''This is an [[#All-script|all-script]]!''
Will effectively run <code>mwscript --wiki <nameofwiki> maintenance/doExample.php [args]</code>.
* for a single file
* checks PHP syntax
* syncs <code>/srv/mediawiki-staging/(some file)</code> -> <code>/srv/mediawiki</code>


=== {{anchor|sync-wikiversions}}<code>scap sync-wikiversions</code>===
===<code>foreachwiki</code>===
'''Note:''' this script is from [https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap <code>mediawiki/tools/scap</code>].
Sequentially run a maintenance script for all wikis in the realm. If run from [[mwmaint1002]], this will use <code>all.dblist</code>. If run from Beta Cluster, it uses <code>all-labs.dblist</code>.
: ''This is an [[#All-script|all-script]]!''
* syncs <code>/srv/mediawiki-staging/wikiversions.{json,cdb}</code> -> <code>/srv/mediawiki</code>


=== {{anchor|scap}}<code>scap sync</code> ===
This is effectively a shortcut for  <code>foreachwikiindblist <all or all-labs> [args]</code> .
{{shortcut|scap}}
[[File:Scap-logo-white-on-black.png|thumb|'scap' logo]]
[[File:Scap-diagram.png|thumb|A diagram of wikimedia 'scap' deployment tool]]
'''Note:''' this script is from <code>[https://git.wikimedia.org/tree/mediawiki%2Ftools%2Fscap mediawiki/tools/scap]</code>. Generated documentation is [https://doc.wikimedia.org/mw-tools-scap/ published on doc.wikimedia.org]
: ''This is an [[#All-script|all-script]]!''
: ''"sync-common-all-php"''
* lints PHP files in <code>./wmf-config</code>, <code>./multiversion</code>
* Sync deploy directory (/srv/mediawiki-staging) on localhost with staging area (/srv/mediawiki)
* for all mediawiki versions currently deployed (usually 2) it:
** rebuilds localization caches of core and extensions
** creates json files from localization caches for better rsync performance
* runs [[#scap_pull|scap pull]] on co-master servers to update <code>/srv/mediawiki-staging</code>
* runs [[#scap_pull|scap pull]] on rsync fanout servers to update <code>/srv/mediawiki</code>
* runs [[#scap_pull|scap pull]] on "all" servers to update <code>/srv/mediawiki</code> from nearest rsync slave
* rebuilds localization caches on "all" servers from json files that were synced by [[#scap_pull|scap pull]]
* compiles wikiversions.json to wikiversions.cdb on localhost
* runs [[#scap_sync-wikiversions|scap sync-wikiversions]]


<syntaxhighlight lang="shell-session">
For example:
bd808@silver:~$ scap say
  foreachwiki maintenance/doExample.php [args]
------------------------------------------------
/                                                \
|    S.C.A.P.: someone can always pontificate    |
\                                                /
------------------------------------------------
    \
    \
      \
          ___ ____
        ⎛  ⎛ ,----
          \  //==--'
    _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
  (∞)_, )  (    |  ______/__ \/ /__ / /_/ / /_/ /
    ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
        ««_/  «_/ jgs/bd808                /_/
</syntaxhighlight>


=== <code>mwgrep</code> ===
=== <code>mwgrep</code> ===
{{shortcut|mwgrep}}
{{shortcut|mwgrep}}
Search for CSS and JavaScript code fragments in MediaWiki, User and Module namespaces pages on all Wikimedia projects (powered by [[Elasticsearch]]).
Search for a pattern in the revision text of all Wikimedia projects, specifically for code-pages with JS or CSS format in the "MediaWiki", "User" or "Module" namespace. (Powered by [[Elasticsearch]]).


It's located in [https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/scap/files/mwgrep operations/puppet/modules/scap/files/mwgrep]
It's located in [https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/scap/files/mwgrep.py operations/puppet/modules/scap/files/mwgrep]


== All-script ==
See also [https://global-search.toolforge.org/ global-search.toolforge.org], a public tool with more-or-less equivalent functionality.
An '''all-script''' is a script that operates on mediawiki-installation boxes via ssh. ''These scripts performs the action on '''all''' servers.''


The specific servers include:
=== <code>expanddblist</code> ===
* [[app servers]]
For convenience, this utility is also available locally as <code>./bin/expanddblist</code> in any checkout of <code>operations/mediawiki-config.git</code>.
* [[image scalers]]
Usage: expanddblist [<dblist>, <dblistname> or <expression>]
* misc other hosts running apache:
Example:
** live html snapshot browsers
./bin/expanddblist flow_computed > dblists/flow.dblist
** ''etc.''
Advanced usage example using <code>expanddblist</code> to parallelise <code>foreachwiki</code>:
/usr/local/bin/expanddblist all | xargs -I{} -P 4 sh -c \'/usr/local/bin/mwscript extensions/Example/maintenance/updateThing.php --wiki={} > /var/log/mediawiki/example-thing/{}.log 2>&1


== Examples ==
=== <code>sql</code> ===
=== <code>sync-apache</code> example ===
{{shortcut|sql}}
# On [[tin]]:
See [[Debugging in production#Debugging databases]].
# (optionally) run:<br/><code>sync-apache-simulated</code><br/>This just has an "-n" added to the [[rsync]] commands. (<code>--dry-run | </code> show what would have been transferred)
# If that looks fine, actually sync:<br/><code>sync-apache</code>
# Connect to a single random "srv" server, confirm the expected change has arrived here, and restart [[Apache]] to check your change did not break things.
## (how to restart that apache?)
# If that also looks fine, restart all Apaches:<br/><code>apache-graceful-all</code>


After this you might want to [[Purge URL|purge a URL]] from [[Squid]] cache. Also see [[Squids#Purge a given external (non WMF) url]].
== Purge ==
Sometimes you may need to [[Purge URL|purge a URL]] from Varnish cache.


In most cases this is enough. However if the change involves more than a simple change to redirects, [[depooling]]/ repooling may be necessary. Make sure you find out whether this is needed ahead of time{{#ifexist: depooling ||&nbsp;(while at it, start the [[depooling]] page and link to it, we need documentation!)}}.
In most cases this is enough. However if the change involves more than a simple change to redirects, [[depooling]]/ repooling may be necessary. Make sure you find out whether this is needed ahead of time{{#ifexist: depooling ||&nbsp;(while at it, start the [[depooling]] page and link to it, we need documentation!)}}.
Line 115: Line 83:
* [[Heterogeneous deployment]]
* [[Heterogeneous deployment]]


[[Category:Software]]
[[Category:MediaWiki production]]
[[Category:How-To]]
[[Category:How-To]]
[[Category:Wikimedia infrastructure]]

Revision as of 03:22, 6 August 2022

The following executable binaries are maintained in version control (operations/puppet:/modules/scap/files and others) and provisioned by Puppet into /usr/local/bin.

Deployment host

These commands on the deployment host (deployment.eqiad.wmnet).

httpbb

View source in gerrit.

mwlog-server

These commands must be run from mwlog1002.

logspam-watch

Periodically refreshes and offers a filtering interface to logspam, which in turn aggregates various PHP error and exception logs.

fatalmonitor

No longer exists - see logspam-watch instead.

Maintenance host

These commands must be run on a maintenance host (e.g. mwmaint1002).

Consider using screen or tmux before starting any long-running maintenance scripts.

mwscript

Maintenance script runner. This wrapper is mandatory for all maintenance scripts in production. It takes care of finding where the appropriate MediaWiki installation on the server for the selected dbname (per Multiversion), setting environment variables, sudo-ing as the correct shell user, and more. See Heterogeneous deployment for more about Multiversion.

Script names must be relative to a MediaWiki install path, for example:

mwscript maintenance/doExample.php --wiki <nameofwiki> [args]

or

mwscript extensions/Example/doExample.php --wiki <nameofwiki> [args]

mwscriptwikiset

Sequentially run a maintenance script for all wikis in a dblist-file (in wmf-config/dblists). For example:

mwscriptwikiset maintenance/doExample.php group0.dblist [args]

This only supports expanded or plain dblist files. It does not support dblist files containing a computable expression.

It effectively runs mwscript maintenance/doExample.php <nameofwiki> [args].

foreachwikiindblist

Sequentially run a maintenance script for all wikis in a dblist expression.

Similar to mwscriptwikiset, except that it supports dblist expressions, and invokes mwscript with --wiki instead of position arguments.

For example:

foreachwikiindblist <dblist or expression> maintenance/doExample.php [args]

Will effectively run mwscript --wiki <nameofwiki> maintenance/doExample.php [args].

foreachwiki

Sequentially run a maintenance script for all wikis in the realm. If run from mwmaint1002, this will use all.dblist. If run from Beta Cluster, it uses all-labs.dblist.

This is effectively a shortcut for foreachwikiindblist <all or all-labs> [args] .

For example:

foreachwiki maintenance/doExample.php [args]

mwgrep

Search for a pattern in the revision text of all Wikimedia projects, specifically for code-pages with JS or CSS format in the "MediaWiki", "User" or "Module" namespace. (Powered by Elasticsearch).

It's located in operations/puppet/modules/scap/files/mwgrep

See also global-search.toolforge.org, a public tool with more-or-less equivalent functionality.

expanddblist

For convenience, this utility is also available locally as ./bin/expanddblist in any checkout of operations/mediawiki-config.git.

Usage: expanddblist [<dblist>, <dblistname> or <expression>]

Example:

./bin/expanddblist flow_computed > dblists/flow.dblist

Advanced usage example using expanddblist to parallelise foreachwiki:

/usr/local/bin/expanddblist all | xargs -I{} -P 4 sh -c \'/usr/local/bin/mwscript extensions/Example/maintenance/updateThing.php --wiki={} > /var/log/mediawiki/example-thing/{}.log 2>&1

sql

See Debugging in production#Debugging databases.

Purge

Sometimes you may need to purge a URL from Varnish cache.

In most cases this is enough. However if the change involves more than a simple change to redirects, depooling/ repooling may be necessary. Make sure you find out whether this is needed ahead of time.

See also