You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Switch Datacenter"

From Wikitech-static
Jump to navigation Jump to search
imported>Giuseppe Lavagetto
imported>BBlack
Line 6: Line 6:
=== Schedule for 2017 switch ===
=== Schedule for 2017 switch ===
See [[phab:T138810]] for tasks to be undertaken during the switch
See [[phab:T138810]] for tasks to be undertaken during the switch
* Deployment server:
* Traffic: Tuesday, April 18th 2017
* Traffic:  
* Elasticsearch: elasticsearch is automatically following mediawiki switch
* MediaWiki 5-minute read-only test: (?)
* Media storage/Swift: Tuesday, April 18th 2017
* ElasticSearch:  
* Media storage/Swift:  
* Services: Tuesday, April 18th 2017
* Services: Tuesday, April 18th 2017
* MediaWiki: '''Wednesday, April 19th 2017''' (requires read-only mode)
* MediaWiki: '''Wednesday, April 19th 2017 14:00 UTC''' (user visible, requires read-only mode)
* Deployment server: (sometime after the switch)


==== Switching back ====
==== Switching back ====
* MediaWiki:  '''Wednesday, May 3rd 2017''' (requires read-only mode)
* Traffic: Pre-switchback in two phases: Mon May 1 and Tues May 2 (to avoid cold-cache issues Weds)
* Services, ElasticSearch, Traffic, Swift, Deployment server: Thursday, May 4th 2017 (after the above is done)
* MediaWiki:  '''Wednesday, May 3rd 2017 14:00 UTC''' (user visible, requires read-only mode)
* Services, Elasticsearch, Swift, Deployment server: Thursday, May 4th 2017 (after the above is done)


=== Schedule for Q3 FY2015-2016 rollout ===
=== Schedule for Q3 FY2015-2016 rollout ===
Line 22: Line 22:
* Traffic: Thursday, March 10th 2016
* Traffic: Thursday, March 10th 2016
* MediaWiki 5-minute read-only test: Tuesday, March 15th 2016, 07:00 UTC
* MediaWiki 5-minute read-only test: Tuesday, March 15th 2016, 07:00 UTC
* ElasticSearch: Thursday, April 7th 2016, 12:00 UTC
* Elasticsearch: Thursday, April 7th 2016, 12:00 UTC
* Media storage/Swift: Thursday, April 14th 2016, 17:00 UTC
* Media storage/Swift: Thursday, April 14th 2016, 17:00 UTC
* Services: Monday, April 18th 2016, 10:00 UTC
* Services: Monday, April 18th 2016, 10:00 UTC
Line 29: Line 29:
==== Switching back ====
==== Switching back ====
* MediaWiki: Thursday, April 21st 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)
* MediaWiki: Thursday, April 21st 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)
* Services, ElasticSearch, Traffic, Swift, Deployment server: Thursday, April 21st 2016, after the above is done
* Services, Elasticsearch, Traffic, Swift, Deployment server: Thursday, April 21st 2016, after the above is done


<br>
<br>
Line 42: Line 42:
# (days in advance) Prepare puppet patches:
# (days in advance) Prepare puppet patches:
#* Switch mw_primary [https://gerrit.wikimedia.org/r/346321]
#* Switch mw_primary [https://gerrit.wikimedia.org/r/346321]
#* Add direct route to $new_site for all mw-related cache::app_directors [https://gerrit.wikimedia.org/r/346320]
#* Switch cache::app_routes backends from old_site-active to new_site-active [https://gerrit.wikimedia.org/r/346320]
#* Comment-out direct route to $old_site in all mw-related cache::app_directors [https://gerrit.wikimedia.org/r/346322]
# (days in advance) Prepare the mediawiki-config patch or patches ([https://gerrit.wikimedia.org/r/346251 example])
# Disable puppet on all jobqueues/videoscalers and maintenance hosts, and the varnishes
# Disable puppet on all jobqueues/videoscalers and maintenance hosts
# Merge the mediawiki-config [https://gerrit.wikimedia.org/r/346251 switchover changes] but don't sync '''This is not covered by the switchdc script'''
# Merge the mediawiki-config [https://gerrit.wikimedia.org/r/346251 switchover changes] but don't sync '''This is not covered by the switchdc script'''
# Reduce the TTL on <tt>appservers-rw, api-rw, imagescaler-rw</tt> to 10 seconds
# Reduce the TTL on <tt>appservers-rw, api-rw, imagescaler-rw</tt> to 10 seconds
Line 68: Line 68:
#* The global warmup against the appservers cluster
#* The global warmup against the appservers cluster
#* The apc-warmup against all hosts in the appservers and api clusters at least.
#* The apc-warmup against all hosts in the appservers and api clusters at least.


==== Phase 5 - switch active datacenter configuration ====
==== Phase 5 - switch active datacenter configuration ====
# Merge the switch of <tt>$mw_primary</tt> at this point and add direct route to $new_site for all mw-related cache::app_directors. both changes can be puppet-merged toghether. '''This is not covered by the switchdc script'''. (Puppet is only involved in managing traffic, db alerts, and the jobrunners).
# Send the traffic layer to active-active:
#* disable puppet on cache::text in both datacenters
#* merge the varnish patch '''This is not covered by the switchdc script'''
#* enable and run puppet on cache::text in $new_site. This starts the active-active traffic phase (traffic will  go to both
# Merge the switch of <tt>$mw_primary</tt> at this point. This change can actually be puppet-merged together with the varnish one. '''This is not covered by the switchdc script'''. (Puppet is only involved in managing traffic, db alerts, and the jobrunners).
# Switch the discovery
# Switch the discovery
#* Flip <tt>appservers-rw, api-rw, imagescaler-rw</tt> to <tt>pooled=true</tt> in the new site. This will not actually change the DNS records, but the on-disk redis config will change.
#* Flip <tt>appservers-rw, api-rw, imagescaler-rw</tt> to <tt>pooled=true</tt> in the new site. This will not actually change the DNS records, but the on-disk redis config will change.
Line 78: Line 83:
==== Phase 6 - apply configuration ====
==== Phase 6 - apply configuration ====
# Switch the live redis configuration. This can be either scripted, or all redises can be restarted (first in the new site, then in the old one). Verify redises are indeed replicating correctly.
# Switch the live redis configuration. This can be either scripted, or all redises can be restarted (first in the new site, then in the old one). Verify redises are indeed replicating correctly.
# Run puppet on the text caches in $new_site and $old_site. '''This starts the PII leak''' [TODO: check with traffic for the whole procedure]
# Run puppet on the text caches in $old_site. This ends the active-active phase.


==== Phase 7 - Set new site's databases to read-write ====
==== Phase 7 - Set new site's databases to read-write ====
Line 92: Line 97:
# Update tendril for new database masters
# Update tendril for new database masters
# Set the TTL for the DNS records to 300 seconds again.
# Set the TTL for the DNS records to 300 seconds again.
# Varnish final reconfiguration:
#* Merge the second traffic puppet patch, comment-out direct route to $old_site in all mw-related cache::app_directors. '''This is not covered by the switchdc script'''
#* Run puppet on all the cache nodes in $old_site '''this ends the PII leak'''
# [Optional] Run the script to fix broken wikidata entities on the maintenance host of the active datacenter: <code>sudo -u www-data mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki --force</code> '''This is not covered by the switchdc script'''
# [Optional] Run the script to fix broken wikidata entities on the maintenance host of the active datacenter: <code>sudo -u www-data mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki --force</code> '''This is not covered by the switchdc script'''


Line 120: Line 122:
=== ElasticSearch ===
=== ElasticSearch ===


Point CirrusSearch to codfw by editing wmgCirrusSearchDefaultCluster [https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L16025-L16027 InitialiseSettings.php]. The usual default value is "local", which means that if mediawiki switches DC, everything should be automatic. For this specific switch, the value has been set to "codfw" to switch Elasticsearch ahead of time.
CirrusSearch talks by default to the local datacenter ($wmfDatacenter). If Mediawiki switches datacenter, elasticsearch will automatically follow.
 
Manually switching CirrusSearch to a specific datacenter can always be done. Point CirrusSearch to codfw by editing wmgCirrusSearchDefaultCluster [https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L16025-L16027 InitialiseSettings.php].


To ensure coherence in case of lost updates, a reindex of the pages modified during the switch can be done by following [[Search#Recovering_from_an_Elasticsearch_outage.2Finterruption_in_updates|Recovering from an Elasticsearch outage / interruption in updates]].
To ensure coherence in case of lost updates, a reindex of the pages modified during the switch can be done by following [[Search#Recovering_from_an_Elasticsearch_outage.2Finterruption_in_updates|Recovering from an Elasticsearch outage / interruption in updates]].

Revision as of 16:12, 5 April 2017

Introduction

A datacenter switchover (from eqiad to codfw, or vice-versa) comprises switching over multiple different components, some of which can happen independently and many of which need to happen in lockstep. This page documents all the steps needed to switch over from a master datacenter to another one, broken up by component.

Schedule for 2017 switch

See phab:T138810 for tasks to be undertaken during the switch

  • Traffic: Tuesday, April 18th 2017
  • Elasticsearch: elasticsearch is automatically following mediawiki switch
  • Media storage/Swift: Tuesday, April 18th 2017
  • Services: Tuesday, April 18th 2017
  • MediaWiki: Wednesday, April 19th 2017 14:00 UTC (user visible, requires read-only mode)
  • Deployment server: (sometime after the switch)

Switching back

  • Traffic: Pre-switchback in two phases: Mon May 1 and Tues May 2 (to avoid cold-cache issues Weds)
  • MediaWiki: Wednesday, May 3rd 2017 14:00 UTC (user visible, requires read-only mode)
  • Services, Elasticsearch, Swift, Deployment server: Thursday, May 4th 2017 (after the above is done)

Schedule for Q3 FY2015-2016 rollout

  • Deployment server: Wednesday, January 20th 2016
  • Traffic: Thursday, March 10th 2016
  • MediaWiki 5-minute read-only test: Tuesday, March 15th 2016, 07:00 UTC
  • Elasticsearch: Thursday, April 7th 2016, 12:00 UTC
  • Media storage/Swift: Thursday, April 14th 2016, 17:00 UTC
  • Services: Monday, April 18th 2016, 10:00 UTC
  • MediaWiki: Tuesday, April 19th 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)

Switching back

  • MediaWiki: Thursday, April 21st 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)
  • Services, Elasticsearch, Traffic, Swift, Deployment server: Thursday, April 21st 2016, after the above is done


Per-service switchover instructions

MediaWiki

We divide the process in logical phases that should be executed sequentially. Within any phase, top-level tasks can be executed in parallel to each other, while subtasks are to be executed sequentially to each other. The phase number is referred to in the names of the tasks in operations/switchdc [1]

Phase 0 - preparation

  1. (days in advance) Warm up databases; see MariaDB/buffer_pool_dump.
  2. (days in advance) Prepare puppet patches:
    • Switch mw_primary [2]
    • Switch cache::app_routes backends from old_site-active to new_site-active [3]
  3. (days in advance) Prepare the mediawiki-config patch or patches (example)
  4. Disable puppet on all jobqueues/videoscalers and maintenance hosts
  5. Merge the mediawiki-config switchover changes but don't sync This is not covered by the switchdc script
  6. Reduce the TTL on appservers-rw, api-rw, imagescaler-rw to 10 seconds

Phase 1 - stop maintenance

  1. Stop jobqueues in the active site
  2. Kill all the cronjobs on the maintenance host in the active site

Phase 2 - read-only mode

  1. Go to read-only mode by syncing wmf-config/db-$old-site.php

Phase 3 - lock down database masters

  1. Put old-site core DB masters (shards: s1-s7, x1, es2-es3) in read-only mode.
  2. Wait for the new site's databases to catch up replication

Phase 4.1 - Wipe caches

  1. Wipe new site's memcached to prevent stale values — only once the new site's read-only master/slaves are caught up.
  2. Restart all HHVM servers in the new site to clear the APC cache

Phase 4.2 - Warmup caches in the new site

This phase will be executed by the t04_cache_wipe task of switchdc, because there is no speed gain from not doing all of phase 4.1 + phase 4.2 separately, and they are logically related.

  1. Warm up memcached and APC running the mediawiki-cache-warmup on the new site clusters, specifically:
    • The global warmup against the appservers cluster
    • The apc-warmup against all hosts in the appservers and api clusters at least.


Phase 5 - switch active datacenter configuration

  1. Send the traffic layer to active-active:
    • disable puppet on cache::text in both datacenters
    • merge the varnish patch This is not covered by the switchdc script
    • enable and run puppet on cache::text in $new_site. This starts the active-active traffic phase (traffic will go to both
  2. Merge the switch of $mw_primary at this point. This change can actually be puppet-merged together with the varnish one. This is not covered by the switchdc script. (Puppet is only involved in managing traffic, db alerts, and the jobrunners).
  3. Switch the discovery
    • Flip appservers-rw, api-rw, imagescaler-rw to pooled=true in the new site. This will not actually change the DNS records, but the on-disk redis config will change.
    • Deploy wmf-config/ConfigSettings.php changes to switch the datacenter in MediaWiki
    • Flip appservers-rw, api-rw, imagescaler-rw to pooled=false in the old site. After this, DNS will be changed and internal applications will start hitting the new DC

Phase 6 - apply configuration

  1. Switch the live redis configuration. This can be either scripted, or all redises can be restarted (first in the new site, then in the old one). Verify redises are indeed replicating correctly.
  2. Run puppet on the text caches in $old_site. This ends the active-active phase.

Phase 7 - Set new site's databases to read-write

  1. Set new-site's core DB masters (shards: s1-s7, x1, es2-es3) in read-write mode.

Phase 8 - Set MediaWiki to read-write

  1. Deploy mediawiki-config wmf-config/db-$new-site.php with all shards set to read-write

Phase 9 - post read-only

  1. Start the jobqueue in the new site by running puppet there (mw_primary controls it)
  2. Run puppet on the maintenance hosts (mw_primary controls it)
  3. Update DNS records for new database masters
  4. Update tendril for new database masters
  5. Set the TTL for the DNS records to 300 seconds again.
  6. [Optional] Run the script to fix broken wikidata entities on the maintenance host of the active datacenter: sudo -u www-data mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki --force This is not covered by the switchdc script

Phase 10 - verification and troubleshooting

  1. Make sure reading & editing works! :)
  2. Make sure recent changes are flowing (see Special:RecentChanges, EventStreams, RCStream and the IRC feeds)
  3. Make sure email works (exim4 -bp on mx1001/mx2001, test an email)

Media storage/Swift

Ahead of the switchover, originals and thumbs

  1. Cache->app: Change varnish backends for swift and swift_thumbs to point to new site with https://gerrit.wikimedia.org/r/#/c/282890/https://gerrit.wikimedia.org/r/284651
    1. Force a puppet run on cache_upload in both sites: salt -v -t 10 -b 17 -C 'G@cluster:cache_upload and ( G@site:eqiad or G@site:codfw )' cmd.run 'puppet agent --test'
  2. Inter-Cache: Switch new site from active site to 'direct' in cache::route_table for upload https://gerrit.wikimedia.org/r/#/c/282891/https://gerrit.wikimedia.org/r/284650
    1. Force a puppet run on cache_upload in new site: salt -v -t 10 -b 17 -C 'G@cluster:cache_upload and G@site:codfw' cmd.run 'puppet agent --test'
  3. Users: De-pool active site in GeoDNS https://gerrit.wikimedia.org/r/#/c/283416/https://gerrit.wikimedia.org/r/#/c/284694/ + authdns-update
  4. Inter-Cache: Switch all caching sites currently pointing from active site to new site in cache::route_table for upload https://gerrit.wikimedia.org/r/#/c/283418/https://gerrit.wikimedia.org/r/284649
    1. Force a puppet run on cache_upload in caching sites: salt -v -t 10 -b 17 -C 'G@cluster:cache_upload and G@site:esams' cmd.run 'puppet agent --test'
  5. Inter-Cache: Switch active site from 'direct' to new site in cache::route_table for upload https://gerrit.wikimedia.org/r/#/c/282892/https://gerrit.wikimedia.org/r/284648
    1. Force a puppet run on cache_upload in active site: salt -v -t 10 -b 17 -C 'G@cluster:cache_upload and G@site:eqiad' cmd.run 'puppet agent --test'

Switching back

Repeat the steps above in reverse order, with suitable revert commits

ElasticSearch

CirrusSearch talks by default to the local datacenter ($wmfDatacenter). If Mediawiki switches datacenter, elasticsearch will automatically follow.

Manually switching CirrusSearch to a specific datacenter can always be done. Point CirrusSearch to codfw by editing wmgCirrusSearchDefaultCluster InitialiseSettings.php.

To ensure coherence in case of lost updates, a reindex of the pages modified during the switch can be done by following Recovering from an Elasticsearch outage / interruption in updates.

Traffic

GeoDNS user routing

Inter-Cache routing

Cache->App routing

Specifics for Switchover Test Week

After switching all applayer services we plan to switch successfully, we'll switch user and inter-cache traffic away from eqiad:

  • The Upload cluster will be following similar instructions on the 14th during the Swift switch.
  • Maps and Misc clusters are not participating (low traffic, special issues, validated by the other moves)
  • This leaves just the text cluster to operate on below:
  1. Inter-Cache: Switch codfw from 'eqiad' to 'direct' in cache::route_table for the text cluster.
  2. Users: De-pool eqiad in GeoDNS for the text cluster.
  3. Inter-Cache: Switch esams from 'eqiad' to 'codfw' in cache::route_table for the text cluster.
  4. Inter-Cache: Switch eqiad from 'direct' to 'codfw' in cache::route_table for the text cluster.

Before reversion of applayer services to eqiad, we'll revert the above steps in reverse order to undo them:

  1. Inter-Cache: Switch eqiad from 'codfw' to 'direct' in cache::route_table for all clusters.
  2. Inter-Cache: Switch esams from 'codfw' to 'eqiad' in cache::route_table for all clusters.
  3. Users: Re-pool eqiad in GeoDNS.
  4. Inter-Cache: Switch codfw from 'direct' to 'eqiad' in cache::route_table for all clusters.

Services

All services, are active-active in DNS discovery, apart from restbase, that needs special treatment. The procedure to fail over to one site only is the same for every one of them:

  1. reduce the TTL of the dns discovery records to 10 seconds
  2. If the service is not active-active in varnish, make it active-active
  3. depool the datacenter we're moving away from in confctl / discovery
  4. Make traffic go to the only still active datacenter restoring the active-passive status in cache::app_directors
  5. restore the original TTL

Restbase is a bit of a special case, and needs an additional step, if we're just switching active traffic over and not simulating a complete failover:

  1. pool restbase-async everywhere, then depool restbase-async in the newly active dc, so that async traffic is separated from real-users traffic as much as possible.

Other miscellaneous