Application servers

Wikimedia infrastructure

Data centers

Networking

Global traffic routing

MediaWiki

• Application servers
• PHP 7 and php-fpm
• Dumps
• Memcached
• Mcrouter runbook
• Kafka Job Queue
• Nutcracker
• Envoy
• MediaWiki HTTP cache headers
• Redis
• Parser cache
• Statsd
• MediaWiki On Kubernetes
• MediaWiki and EtcdConfig
• Citoid
• Mathoid

Multimedia

Data Engineering

Search

Data Persistence

Infra Foundations

Observability

Performance Team

[]

See also Application servers/Runbook for how to perform common tasks, or diagnose issues.

The Application servers (or app servers) are the several hundred Apache servers that run the MediaWiki backend software (written in PHP).

Service

Puppet roles:

• mediawiki::appserver, mediawiki::canary_appserver
• mediawiki::appserver::api, mediawiki::appserver::canary_api
• mediawiki::maintenance
• mediawiki::jobrunner

Relevant puppet classes:

• profile::mediawiki::webserver, this provisions Apache, and any other packages or resources needed by MediaWiki on app servers.
  • profile::mediawiki::httpd, the Apache service.
  • meidawiki::web::prod_sites, the Apache configuration for all production websites (including wikipedia.org).
  • Additional Apache configurations are at modules/mediawiki/files/apache/sites/. Prior to 2012, Apache configuration were in a Subversion repository.

Architecture

See also: MediaWiki at WMF and HTTP timeouts#App servers

The application servers are load-balanced via LVS. Connections between our CDN (HTTP cache proxies) and app servers are encrypted with TLS, which is terminated locally on the app server using a simple Nginx- install. Nginx then hands the request off to the local Apache.

Apache there is in charge of handling redirects, rewrite rules, and determining the document root. It then uses php-fpm to invoke the MediaWiki software.

The Apache MPM we use is mod_worker, which decides how php-fpm processes are spawned.

Logging

Apache errors are logged to /srv/mw-log/apache2.log on mwlog1001.

Apache access logs are mostly disabled. Statistics are drawn from Varnish front ends instead.

Apache setup checklist

• Follow the Automated installation instructions for the base install
• Run the following on the server:

• apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit

• Wait for the server to come back online, ensure it starts apache correctly
  • echo 'GET /' | nc localhost 80 or any of the number of tests listed below
• If the server is part of the memcached group, follow instructions on Memcached
• If the server is new, you will need to do the following:

• Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches

• If the server is not new do the following:

• Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches

• You will need to add the server to DSH groups if new, or check if they are commented, if the server is not new:

• Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed
• Reload nagios to accept the changes to the node groups:

• cd /home/wikipedia/conf/nagios && ./sync

• Verify that the server is tacking traffic and doing work

• ipvsadm -L | grep SERVERNAME
• traffic logs?

Test cases

Here are some test cases you can use to test the apache configuration after changing something.

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: www.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: www.wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikibooks.org
User-agent: testthing

GET / HTTP/1.1
Host: wikiquote.org
User-agent: testthing

GET / HTTP/1.1
Host: dk.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: foo.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: test.wikipedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-Agent: Exalead

GET /wiki/Foo HTTP/1.1
Host: meta.wikimedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en.wiktionary.org
User-agent: testthing

Hardware repair

When taking down application servers (running mediawiki) for things like disk replacement or other hardware repair, _do not forget to_:

• before: remove from dsh group

These are in puppet, operations/puppet repo, in modules/dsh/files/group. The important one for Mediawiki sync is "mediawiki-installation".

• before: de-pool in pybal
• TODO: Document what to do if it's a scap proxy (see hieradata/common/dsh/config.yaml)

See pybal. You can just grep for the server name and set 'enabled': False and save.

• before: check nobody is scapping right now (best: announce with a !log line in IRC)

This is an IRC thing on freenode in #wikimedia-dev/-tech/-operations

• during: acknowledge Icinga monitoring checks (best: with related ticket number as comment)

Do this by logging in via browser on icinga.wikimedia.org. search for the hostname, check all services and use the "acknowledge" option. You'll see the IRC bots outputting this as well and they will stop repeating things over and over in the channels.

• after: re-add to dsh groups

Revert the above.

• after: re-pool in pybal

Revert the above.

See also

• Apache log format
• UID