You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
The Apache configs are both in puppet and in
/home/wikipedia/conf/httpds. This guide mostly covers the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track changes.
It is suggested that you may wish to place any configuration updates on the Deployments page. A bad configuration going live can easily result in a site outage.
- Submit change to gerrit in the
modules/mediawiki/files/apache/sitesdirectory (project: operations/puppet)
- disable puppet across the mw-cluster (so you can test the change to a single host):
salt --batch-size=25% 'mw*' cmd.run 'puppet agent --disable'
- Merge via gerrit
- on palladium:
check if the config does not break Apache! You can do this by just disabling puppet on the appservers, run the puppet tagged run to a single server and restarting that manually before pushing out to all. Use a test script to check multiple URLs to see if redirect changes work.
- go to one server and manually re-enable puppet & run puppet
- confirm the configuration changes go through
- go to one mw server and do
- create a plain text file with some significant URLs you touched and use "apache-fast-test url.file mw1234" on tin to test against your one test host.
- once test works, re-enable puppet across the mw hosts:
salt --batch-size=25% 'mw*' cmd.run 'puppet agent --enable'
- sync the code via a tagged puppet run on appservers:
salt --batch-size=25% 'mw*' cmd.run 'puppet agent -t --tags mw-apache-config'
- Test change from external
Connect to fenari and type apache-graceful-all. This will restart all apache processes for servers in the apaches dsh node group. You must be root to do this.. <-- fenari doesn't exist anymore, neither does the script. use Salt instead from a salt master like palladium.
One, to test a change
Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type apache-graceful to restart apache on that web server only. Test your change with curl, as with this foundation example, with what you type in italics:
curl -H 'Host: wikimediafoundation.org' "http://localhost/fundraising"
The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.
Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.
Apache access logs are mostly disabled. Webalizer statistics are made using the squids logs instead.
Text output on stderr from various external commands is in various log files in /home/wikipedia/logs on fenari. PHP errors are in /home/wikipedia/logs/syslog/syslog on fenari.
Apache setup checklist
- Follow the Automated installation instructions for the base install
- Run the following on the server:
- apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit
- Wait for the server to come back online, ensure it starts apache correctly
- echo 'GET /' | nc localhost 80 or any of the number of tests listed below
- If the server is part of the memcached group, follow instructions on Memcached
- Run the setup of Ganglia
- If the server is new, you will need to do the following:
- Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
- If the server is not new do the following:
- Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
- You will need to add the server to DSH groups if new, or check if they are commented, if the server is not new:
- Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
- Reload nagios to accept the changes to the node groups:
- cd /home/wikipedia/conf/nagios && ./sync
- Verify that the server is tacking traffic and doing work
- ipvsadm -L | grep SERVERNAME
- traffic logs?
Here are some test cases you can use to test the apache configuration after changing something.
GET /wiki/Foo HTTP/1.1 Host: en.wikipedia.org User-agent: testthing GET /wiki/Foo HTTP/1.1 Host: www.wikipedia.org User-agent: testthing GET / HTTP/1.1 Host: en2.wikipedia.org User-agent: testthing GET /wiki/Main_Page HTTP/1.1 Host: www.wikipedia.com User-agent: testthing GET / HTTP/1.1 Host: wikipedia.com User-agent: testthing GET / HTTP/1.1 Host: wikibooks.org User-agent: testthing GET / HTTP/1.1 Host: wikiquote.org User-agent: testthing GET / HTTP/1.1 Host: dk.wikipedia.org User-agent: testthing GET / HTTP/1.1 Host: foo.wikipedia.org User-agent: testthing GET /wiki/Main_Page HTTP/1.1 Host: test.wikipedia.org User-agent: testthing GET / HTTP/1.1 Host: webshop.wikipedia.org User-agent: testthing GET / HTTP/1.1 Host: boards.wikimedia.org User-agent: testthing GET /wiki/Foo HTTP/1.1 Host: en.wikipedia.org User-Agent: Exalead GET /wiki/Foo HTTP/1.1 Host: meta.wikimedia.org User-agent: testthing GET / HTTP/1.1 Host: en.wiktionary.org User-agent: testthing
When taking down application servers (running mediawiki) for things like disk replacement or other hardware repair, _do not forget to_:
- before: remove from dsh groups
These are in puppet, operations/puppet repo, in files/dsh/group. The important one for Mediawiki sync is "mediawiki-installation". The important one for Apache config sync is "apaches". Make sure those 2 are edited but also grep through all, there will be others where it's unclear if they are used.
- before: de-pool in pybal
- before: check nobody is scapping right now (best: announce with a !log line in IRC)
This is an IRC thing on freenode in #wikimedia-dev/-tech-/-operations
- during: acknowledge Icinga monitoring checks (best: with related RT ticket number as comment)
Do this by logging in via browser on icinga-admin.wikimedia.org. search for the hostname, check all services and use the "acknowledge" option. You'll see the IRC bots outputting this as well and they will stop repeating things over and over in the channels.
- after: re-add to dsh groups
Revert the above.
- after: re-pool in pybal
Revert the above.