Application servers: Difference between revisions

Revision as of 02:30, 1 May 2022

See also Application servers/Runbook for how to perform common tasks, or diagnose issues.

The Application servers (or app servers) are the several hundred Apache servers that run MediaWiki (PHP application).

Service

Puppet roles:

mediawiki::appserver, mediawiki::canary_appserver
mediawiki::appserver::api, mediawiki::appserver::canary_api
mediawiki::maintenance
mediawiki::jobrunner

Relevant puppet classes:

profile::mediawiki::webserver, this provisions Apache, and any other packages or resources needed by MediaWiki on app servers.
- profile::mediawiki::httpd, the Apache service.
- mediawiki::web::prod_sites, the Apache configuration for all production websites (including wikipedia.org).
- Additional Apache configurations are at modules/mediawiki/files/apache/sites/. Prior to 2012, Apache configuration were in a Subversion repository.

Architecture

The application servers are load-balanced via LVS. Connections between our CDN (HTTP cache proxies) and app servers are encrypted with TLS, which is terminated locally on the app server using Envoy. Envoy then hands the request off to the local Apache.

Apache is in charge of handling redirects, rewrite rules, and determining the document root. It then uses php-fpm to invoke the MediaWiki software.

The Apache MPM we use is mod_worker, which decides how php-fpm processes are spawned.

Logging

Apache errors are logged to /srv/mw-log/apache2.log on mwlog1001.

Apache access logs are mostly disabled. Statistics are drawn from Varnish front ends instead.

Hardware repair

When taking down application servers (running mediawiki) for things like disk replacement or other hardware repair, _do not forget to_:

before: remove from dsh group

These are in puppet, operations/puppet repo, in modules/dsh/files/group. The important one for Mediawiki sync is "mediawiki-installation".

before: de-pool in pybal
TODO: Document what to do if it's a scap proxy (see hieradata/common/dsh/config.yaml)

See pybal. You can just grep for the server name and set 'enabled': False and save.

before: check nobody is scapping right now (best: announce with a !log line in IRC)

This is an IRC thing on libera.chat in #wikimedia-dev ^connect/#wikimedia-tech ^connect/#wikimedia-operations ^connect

during: acknowledge Icinga monitoring checks (best: with related ticket number as comment)

Do this by logging in via browser on icinga.wikimedia.org. search for the hostname, check all services and use the "acknowledge" option. You'll see the IRC bots outputting this as well and they will stop repeating things over and over in the channels.

after: re-add to dsh groups

Revert the above.

after: re-pool in pybal

Revert the above.

@@ Line 1: / Line 1: @@
-The Apache configurations are maintained in a Git repository at [https://github.com/wikimedia/operations-puppet/tree/production/modules/mediawiki/files/apache/sites operations/puppet.git:/modules/mediawiki/files/apache/sites/]. Before July 2012 these were in Subversion.
+{{Navigation Wikimedia infrastructure|expand=mw}}
+{{See|See also '''[[Application servers/Runbook]]''' for how to perform common tasks, or diagnose issues.}}
-==Testing config==
+The '''Application servers''' (or '''app servers''') are the several hundred Apache servers that run [[MediaWiki]] (PHP application).
-* Submit change to [[Gerrit]] in the <code>modules/mediawiki/files/apache/sites</code> directory (project: operations/puppet)
-* Disable puppet on [[X-Wikimedia-Debug#Available_backends|one of the mediawiki debug servers]] (assuming you choose <code>mwdebug1001</code>): <code>sudo puppet agent --disable 'insert reason'</code>
-* Apply change locally under <code>/etc/apache2/sites-enabled/</code>
-* On <code>mwdebug1001</code>: <code>sudo apache2ctl restart</code>
-* Test your change by making relevant HTTP request. See [[Debugging in production#Debugging a web request|Debugging in production]] for how.
-* When you're done, <code>sudo puppet agent --enable</code>
-==Deploying config==
+==Service==
-It is suggested that you may wish to place any configuration updates on the [[Deployments]] page.  A bad configuration going live can easily result in a site outage.
+Puppet roles:
-* Test your change in deployment-prep and make sure that it works as expected.
+* <code>mediawiki::appserver</code>, <code>mediawiki::canary_appserver</code>
+* <code>mediawiki::appserver::api</code>, <code>mediawiki::appserver::canary_api</code>
+* <code>mediawiki::maintenance</code>
+* <code>mediawiki::jobrunner</code>
-* Submit change to [[gerrit]] in the <code>modules/mediawiki/files/apache/sites</code> directory (project: operations/puppet)
+Relevant puppet classes:
-* Disable puppet across the affected mediawiki application servers.
+* <code>[https://gerrit.wikimedia.org/g/operations/puppet/+/HEAD/modules/profile/manifests/mediawiki/webserver.pp profile::mediawiki::webserver]</code>, this provisions Apache, and any other packages or resources needed by MediaWiki on app servers.
-** Cumin can in finding the precise set of hosts. For example, this is a recent query: <syntaxhighlight lang="bash">
+** <code>[https://gerrit.wikimedia.org/g/operations/puppet/+/HEAD/modules/profile/manifests/mediawiki/httpd.pp profile::mediawiki::httpd]</code>, the Apache service.
-cumin 'R:File = "/etc/apache2/sites-available/04-remnant.conf"' 'disable-puppet "elukey - precaution for https://gerrit.wikimedia.org/r/#/c/380774/"' -b 10
+** <code>[https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/HEAD/modules/mediawiki/manifests/web/prod_sites.pp mediawiki::web::prod_sites]</code>, the Apache configuration for all production websites (including wikipedia.org).
-</syntaxhighlight> In this case the change was related to a RewriteRule change in '''04-remnant.conf''', but of course it must be changed every time with the file(s) modified by the Gerrit change.
+** Additional Apache configurations are at [https://github.com/wikimedia/operations-puppet/tree/production/modules/mediawiki/files/apache/sites modules/mediawiki/files/apache/sites/]. Prior to 2012, Apache configuration were in a Subversion repository.
-* Merge via gerrit and run on puppetmaster1001 the usual <code>puppet-merge</code>
-* Create a plain text file with some significant URLs that should be modified by the Gerrit change. An example is the de-facto testing authority '''/home/oblivian/baseurls''' on tin. This text file will be used on tin with '''apache-fast-test''' later on to verify that  the change works as expected.
+==Architecture==
-** For example, if you are adding or modifying a new RewriteRule, please add to your text file some URLs that are expected to change.
+{{See also|MediaWiki at WMF|HTTP timeouts#App servers}}
-* Go to one of the '''mwdebug''' servers and enable/run puppet. Apache will reload its configuration automatically, please check that no error messages are emitted. Running '''apachectl -t''' after running puppet surely helps verifying that the new configuration is syntactically correct (it doesn't absolutely imply that it will work as intended of course).
-** Some Apache directive changes need a full restart to get applied, not a simple reload. These changes are very rare and they are clearly indicated in Apache's documentation, so please verify it beforehand. Simple RewriteRule changes require only an Apache reload.
-* On tin run '''apache-fast-test''' against the selected '''mwdebug''' host using '''/home/oblivian/baseurls and your new test file'''. '''Both of them need to return a positive confirmation that everything looks good.'''
-** Example of usage related to the previously mentioned change (https://gerrit.wikimedia.org/r/#/c/380774):
-<syntaxhighlight lang="bash">
-elukey@tin:~$ apache-fast-test /home/oblivian/baseurls mwdebug1002.eqiad.wmnet
-testing 19 urls on 1 servers, totalling 19 requests
-spawning threads..
-http://elefante-a-pallini.ro.sa/
+The application servers are load-balanced via [[LVS]]. Connections between our CDN (HTTP cache proxies) and app servers are encrypted with TLS, which is terminated locally on the app server using [[Envoy]]. Envoy then hands the request off to the local Apache.
- * 200 OK 929
-http://wikimedia.org/research
- * 301 Moved Permanently https://wikimedia.qualtrics.com/SE/?SID=SV_6R04ammTX8uoJFP
-http://www.wikipedia.org/wiki/it:Francesco_Totti
- * 302 Found http://it.wikipedia.org/wiki/Francesco_Totti
-http://zero.wikipedia.org/
- * 302 Found http://en.zero.wikipedia.org/wiki/Special:ZeroRatedMobileAccess
-[.. cut ..]
- * 301 Moved Permanently https://meta.wikimedia.org/wiki/Special:UrlShortener
+'''Apache''' is in charge of handling redirects, rewrite rules, and determining the [[MediaWiki at WMF#Document root|document root]]. It then uses <code>php-fpm</code> to invoke the MediaWiki software.
-elukey@tin:~$ apache-fast-test wikidata_redirect mwdebug1002.eqiad.wmnet
+The Apache [https://httpd.apache.org/docs/2.4/mpm.html MPM] we use is [https://httpd.apache.org/docs/2.4/mod/worker.html mod_worker]</code>, which decides how <code>php-fpm</code> processes are spawned.
-testing 1 urls on 1 servers, totalling 1 requests
-spawning threads..
-https://commons.wikimedia.org/data/main/Data:Bundestagswahl2017/wahlkreis46.map
- * 301 Moved Permanently https://commons.wikimedia.org/wiki/Special:PageData/main/Data:Bundestagswahl2017/wahlkreis46.map
-</syntaxhighlight>
-* Enable/Run puppet on another mediawiki application server that is taking traffic, de-pooling it beforehand via confctl. Verify again from tin that everything is working as expected, running apache-fast-test.
-* Repool the host mentioned above and verify on Apache access logs that everything looks fine. If you want to be extra paranoid, you can check the host level metrics via https://grafana.wikimedia.org/dashboard/db/prometheus-apache-hhvm-dc-stats and make sure that nothing is out of the ordinary.
-* Re-enable puppet across the appservers previously disabled via cumin.
-* Keep an eye on the operations channel and make sure that puppet runs fine on these hosts.
 ==Logging==
-Apache errors are logged to /a/mw-log/apache2.log on fluorine.
+Apache errors are logged to <code>/srv/mw-log/apache2.log</code> on <code>mwlog1001</code>.
 Apache access logs are mostly disabled. Statistics are drawn from [[Varnish]] front ends instead.
-==Apache setup checklist==
+== Hardware repair ==
+{{Outdated-inline|year=2015}}
-* Follow the [[Automated installation]] instructions for the base install
-* Run the following on the server:
-:* <tt>apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit </tt>
-* Wait for the server to come back online, ensure it starts apache correctly
-** <tt>echo 'GET /' | nc localhost 80</tt> or any of the number of tests listed below
-* If the server is part of the memcached group, follow instructions on [[Memcached]]
-* If the server is new, you will need to do the following:
-:* Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
-* If the server is not new do the following:
-:* Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
-* You will need to add the server to [[DSH]] groups if new, or check if they are commented, if the server is not new:
-:* Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed
-:* Reload nagios to accept the changes to the node groups:
-::* <tt>cd /home/wikipedia/conf/nagios && ./sync </tt>
-* Verify that the server is tacking traffic and doing work
-:* <tt>ipvsadm -L | grep SERVERNAME </tt>
-:* traffic logs?
-==Test cases==
-Here are some test cases you can use to test the apache configuration after changing something.
-<pre>
-GET /wiki/Foo HTTP/1.1
-Host: en.wikipedia.org
-User-agent: testthing
-GET /wiki/Foo HTTP/1.1
-Host: www.wikipedia.org
-User-agent: testthing
-GET /wiki/Main_Page HTTP/1.1
-Host: www.wikipedia.com
-User-agent: testthing
-GET / HTTP/1.1
-Host: wikipedia.com
-User-agent: testthing
-GET / HTTP/1.1
-Host: wikibooks.org
-User-agent: testthing
-GET / HTTP/1.1
-Host: wikiquote.org
-User-agent: testthing
-GET / HTTP/1.1
-Host: dk.wikipedia.org
-User-agent: testthing
-GET / HTTP/1.1
-Host: foo.wikipedia.org
-User-agent: testthing
-GET /wiki/Main_Page HTTP/1.1
-Host: test.wikipedia.org
-User-agent: testthing
-GET /wiki/Foo HTTP/1.1
-Host: en.wikipedia.org
-User-Agent: Exalead
-GET /wiki/Foo HTTP/1.1
-Host: meta.wikimedia.org
-User-agent: testthing
-GET / HTTP/1.1
-Host: en.wiktionary.org
-User-agent: testthing
-</pre>
-== Hardware Repair ==
-==== Application Servers ====
 When taking down application servers (running mediawiki) for things like disk replacement or other hardware repair, _do not forget to_:
 * before: remove from dsh group
@@ Line 146: / Line 42: @@
 See [[pybal]]. You can just grep for the server name and set 'enabled': False and save.
 * before: check nobody is scapping right now (best: announce with a !log line in IRC)
-This is an IRC thing on freenode in #wikimedia-dev/-tech/-operations
+This is an IRC thing on libera.chat in {{irc|wikimedia-dev}}/{{irc|wikimedia-tech}}/{{irc|wikimedia-operations}}
 * during: acknowledge Icinga monitoring checks (best: with related ticket number as comment)
 Do this by logging in via browser on icinga.wikimedia.org. search for the hostname, check all services and use the "acknowledge" option. You'll see the IRC bots outputting this as well and they will stop repeating things over and over in the channels.
@@ Line 159: / Line 55: @@
 [[Category:Servers by usage| Apache]]
+[[Category:MediaWiki production| ]]
+[[Category:SRE Service Operations]]

Application servers: Difference between revisions

Revision as of 02:30, 1 May 2022

Contents

Service

Architecture

Logging

Hardware repair

See also

Navigation menu

Application servers: Difference between revisions

Revision as of 02:30, 1 May 2022

Service

Architecture

Logging

Hardware repair

See also

Navigation menu

Search