You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Switch Datacenter"

From Wikitech-static
Jump to navigation Jump to search
imported>BBlack
(updated traffic time for Apr 18, re-ordered stanza in time order)
imported>Quiddity
(→‎Upcoming and past switches: pluralize sub-section titles, remove month from 2021 subtitle (I got confused looking for the Sept date!))
(39 intermediate revisions by 16 users not shown)
Line 1: Line 1:
{{See|See also "[https://blog.wikimedia.org/2016/04/11/wikimedia-failover-test/ Failover Test]" on the Wikimedia Techblog. (April 11, 2016)}}
{{See|See also the blog posts from the "[https://blog.wikimedia.org/2016/04/11/wikimedia-failover-test/ 2016 Failover]" and "[https://blog.wikimedia.org/2017/04/18/codfw-temporary-editing-pause/ 2017 Failover]" on the Wikimedia Blog.}}


== Introduction ==
== Introduction ==
A datacenter switchover (from eqiad to codfw, or vice-versa) comprises switching over multiple different components, some of which can happen independently and many of which need to happen in lockstep. This page documents all the steps needed to switch over from a master datacenter to another one, broken up by component.
A datacenter switchover (from eqiad to codfw, or vice-versa) comprises switching over multiple different components, some of which can happen independently and many of which need to happen in lockstep. This page documents all the steps needed to switch over from a master datacenter to another one, broken up by component. SRE Service Operations maintains the process and software necessary to run the switchover.


=== Schedule for 2017 switch ===
__TOC__
See [[phab:T138810]] for tasks to be undertaken during the switch


* Elasticsearch: elasticsearch is automatically following mediawiki switch
== Weeks in advance preparation ==
* Services: Tuesday, April 18th 2017 14:30 UTC
* Media storage/Swift: Tuesday, April 18th 2017 15:00 UTC
* Traffic: Tuesday, April 18th 2017 19:00 UTC
* MediaWiki: '''Wednesday, April 19th 2017 14:00 UTC''' (user visible, requires read-only mode)
* Deployment server: (sometime after the switch)


==== Switching back ====
* 10 weeks before: Coordinate dates and communication plan with involved groups: [[Switch_Datacenter/Coordination]]
* Traffic: Pre-switchback in two phases: Mon May 1 and Tues May 2 (to avoid cold-cache issues Weds)
* 3 weeks before: Run a "live test" of the cookbooks by "switching" from the passive DC to the active DC. The <code>--live-test</code> flag will skip actions that could harm the active DC or do them on the passive DC. This will instrument most of the code paths used in the switchover and help identify issues. This process will !log to SAL so you should coordinate with others, but otherwise should be non-disruptive. Due to changes since the last switchover you can expect code changes to become necessary, so take the time and assistance needed into account.
* MediaWiki:  '''Wednesday, May 3rd 2017 14:00 UTC''' (user visible, requires read-only mode)
* Services, Elasticsearch, Swift, Deployment server: Thursday, May 4th 2017 (after the above is done)


=== Schedule for Q3 FY2015-2016 rollout ===
== Overall switchover flow ==
* Deployment server: Wednesday, January 20th 2016
In a controlled switchover we first deactivate services in the primary datacenter and  second deactivate caching in the datacenter. The next step is to switch Mediawiki itself. About a week later we activate caching in the datacenter again, as we believe that testing the situation without caching in the datacenter is sufficient.
* Traffic: Thursday, March 10th 2016
* MediaWiki 5-minute read-only test: Tuesday, March 15th 2016, 07:00 UTC
* Elasticsearch: Thursday, April 7th 2016, 12:00 UTC
* Media storage/Swift: Thursday, April 14th 2016, 17:00 UTC
* Services: Monday, April 18th 2016, 10:00 UTC
* MediaWiki: Tuesday, April 19th 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)


==== Switching back ====
Typical scheduling looks like:
* MediaWiki: Thursday, April 21st 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)
* Monday 14:00 UTC Services
* Services, Elasticsearch, Traffic, Swift, Deployment server: Thursday, April 21st 2016, after the above is done
* Monday 15:00 UTC Caching (traffic)
* Tuesday 14:00 UTC Mediawiki


<br>
The following week reactivate caching. 6+ Weeks later switchback Mediawiki
__TOC__


== Per-service switchover instructions ==
== Per-service switchover instructions ==
=== MediaWiki ===
=== MediaWiki ===
We divide the process in logical phases that should be executed sequentially. Within any phase, top-level tasks can be executed in parallel to each other, while subtasks are to be executed sequentially to each other. The phase number is referred to in the names of the tasks in operations/switchdc [https://github.com/wikimedia/operations-switchdc]
We divide the process in logical phases that should be executed sequentially. Within any phase, top-level tasks can be executed in parallel to each other, while subtasks are to be executed sequentially to each other. The phase number is referred to in the names of the tasks in the [[gerrit:plugins/gitiles/operations/cookbooks/|operations/cookbooks]] repository, in the [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/|cookbooks/sre/switchdc/mediawiki/]] path.
 
 


==== Days in advance preparation ====
==== Days in advance preparation ====
# Warm up databases; see [[MariaDB/buffer pool dump]].
# '''OPTIONAL: SKIP IN AN EMERGENCY''': Make sure databases are in a good state. Normally this requires no operation, as the passive datacenter databases are always prepared to receive traffic, so there are no actionables. Some things that #DBAs normally should make sure for the most optimal state possible (sanity checks):
# Prepare puppet patches:
#* There is no ongoing long-running maintenance that affects database availability or lag (schema changes, upgrades, hardware issues, etc.). Depool those servers that are not ready.
#* Switch mw_primary [https://gerrit.wikimedia.org/r/346321]
#* Replication is flowing from eqiad -> codfw and from codfw -> eqiad (replication is usually stopped in the passive -> active direction to facilitate maintenance)
#* Switch cache::app_routes backends from old_site-active to new_site-active [https://gerrit.wikimedia.org/r/346320]
#* All database servers have its buffer pool filled up. This is taken care automatically with the [[MariaDB/buffer pool dump|automatic buffer pool warmup functionality]]. For sanity checks, some sample load could be sent to the MediaWiki application server to check requests happen as quickly as in the active datacenter.
# Prepare the mediawiki-config patch or patches ([https://gerrit.wikimedia.org/r/346251 example])
#* These were the [[Switch Datacenter/planned db maintenance#2018 Switch Datacenter|things we prepared/checked for the 2018 switch]]
# Make absolutely sure that parsercache replication is working from the active to the passive DC. '''This is important'''
#Check appserver weights on servers in the passive DC, make sure that newer hardware is weighted higher (usually 30) and older hardware is less (usually 25)


==== Stage 0 - preparation ====
==== Phase 0 - preparation ====
# Disable puppet on all MediaWiki jobqueues/videoscalers and maintenance hosts and cache::text in both eqiad and codfw. [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t00_disable_puppet.py switchdc t00_disable_puppet.py] [[Switch Datacenter/MediaWiki#t00 disable puppet|sample output]]
# Disable puppet on maintenance hosts in both eqiad and codfw: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/00-disable-puppet.py|00-disable-puppet.py]]
# Merge the mediawiki-config [https://gerrit.wikimedia.org/r/346251 switchover changes] but don't sync '''This is not covered by the switchdc script'''
# Reduce the TTL on <code>appservers-ro, appservers-rw, api-ro, api-rw, jobrunner, videoscaler, parsoid-php</code> to 10 seconds: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/00-reduce-ttl.py|00-reduce-ttl.py]] The warmup step (below) typically takes longer than the old TTL (5 minutes) so this step doesn't wait it to expire. '''Make sure that at least 5 minutes have passed before moving to Phase 1, the cookbook should force you to wait'''.
# Stop <code>swiftrepl</code> on <code>me-fe1005</code>  '''This is not covered by the switchdc script'''
# Warm up APC running the mediawiki-cache-warmup on the new site clusters. The warmup queries will repeat automatically until the response times stabilize: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/00-warmup-caches.py|00-warmup-caches.py]]  
# Reduce the TTL on <tt>appservers-rw, api-rw, imagescaler-rw</tt> to 10 seconds: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t00_reduce_ttl.py switchdc t00_reduce_ttl.py] [[Switch Datacenter/MediaWiki#t00 reduce ttl|sample output]]
#* The global "urls-cluster" warmup against the appservers cluster
#* The "urls-server" warmup against all hosts in the appservers cluster.
#*The "urls-server" warmup against all hosts in the api-appservers cluster.
# Set downtime for Read only checks on mariadb masters changed on Phase 3 so they don't page. '''This is not covered by the switchdc script.'''


==== Phase 1 - stop maintenance ====
==== Phase 1 - stop maintenance ====
# Stop jobqueues in the ''active site'' and kill all the cronjobs on the maintenance host in the active site: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t01_stop_maintenance.py switchdc t01_stop_maintenance.py] [[Switch Datacenter/MediaWiki#output 2|sample output]]
# Stop maintenance jobs in both datacenters and kill all the periodic jobs (systemd timers and cronjobs) on all maintenance host in both datacenters: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/01-stop-maintenance.py|01-stop-maintenance.py]]


==== Phase 2 - read-only mode ====
==== Phase 2 - read-only mode ====
# Go to read-only mode by syncing <tt>wmf-config/db-$old-site.php</tt>: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t02_start_mediawiki_readonly.py switchdc t02_start_mediawiki_readonly.py] [[Switch Datacenter/MediaWiki#output 3|sample output]]
# Go to read-only mode by changing the <code>ReadOnly</code> conftool value: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/02-set-readonly.py|02-set-readonly.py]]


==== Phase 3 - lock down database masters ====
==== Phase 3 - lock down database masters ====
# Put old-site core DB masters (shards: s1-s7, x1, es2-es3) in read-only mode: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t03_coredb_masters_readonly.py switchdc t03_coredb_masters_readonly.py] [[Switch Datacenter/MediaWiki#output 4|sample output]]
# Put old-site core DB masters (shards: s1-s8, x1, es4-es5) in read-only mode and wait for the new site's databases to catch up replication: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/03-set-db-readonly.py|03-set-db-readonly.py]]
 
==== Phase 4 - switch active datacenter configuration ====


==== Phase 4 - Wipe caches in the new site and warmup them ====
# Switch the discovery records and MediaWiki active datacenter: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/04-switch-mediawiki.py|04-switch-mediawiki.py]]  
# All the following tasks are performed by [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t04_cache_wipe.py switchdc t04_cache_wipe.py] [[Switch Datacenter/MediaWiki#output 5|sample output]]
#* Flip <code>appservers-ro, appservers-rw, api-ro, api-rw, jobrunner, videoscaler, parsoid-php</code> to <code>pooled=true</code> in the new site. Since both sites are now pooled in etcd, this will not actually change the DNS records for the active datacenter.
## Wait for the new site's databases to catch up replication
#* Flip <code>WMFMasterDatacenter</code> from the old site to the new.
## Wipe ''new site''<nowiki/>'s memcached to prevent stale values — only once the new site's read-only master/slaves are caught up.
#* Flip <code>appservers-ro, appservers-rw, api-ro, api-rw, jobrunner, videoscaler, parsoid-php</code> to <code>pooled=false</code> in the old site. After this, DNS will be changed for the old DC and internal applications (except mediawiki) will start hitting the new DC
## Restart all HHVM servers in the new site to clear the APC cache
## Warm up memcached and APC running the mediawiki-cache-warmup on the new site clusters, specifically:
##* The global warmup against the appservers cluster
##* The apc-warmup against all hosts in the appservers cluster.
# Merge and puppet-merge the traffic change for text caches https://gerrit.wikimedia.org/r/#/c/346320/. '''This is not covered by the switchdc script'''..


==== Phase 5 - switch active datacenter configuration ====
==== Phase 5 - Invert Redis replication for MediaWiki sessions ====
# Send the traffic layer to active-active: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t05_switch_traffic.py switchdc t05_switch_traffic.py] [[Switch Datacenter/MediaWiki#t05 switch traffic|sample output]]
#* enable and run puppet on cache::text in $new_site. This starts the active-active traffic phase (traffic will  go to both MW clusters)
#* ensure that the change was applied on all hosts in $new_site
#* Run puppet on the text caches in $old_site. This ends the active-active phase.
# Merge the switch of <tt>$mw_primary</tt> at this point. This change can actually be puppet-merged together with the varnish one. '''This is not covered by the switchdc script'''. (Puppet is only involved in managing traffic, db alerts, and the jobrunners).
# Switch the discovery: switchdc [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t05_switch_datacenter.py t05_switch_datacenter] [[Switch Datacenter/MediaWiki#t05 switch datacenter|sample output]]
#* Flip <tt>appservers-rw, api-rw, imagescaler-rw</tt> to <tt>pooled=true</tt> in the new site. This will not actually change the DNS records, but the on-disk redis config will change.
#* Deploy <tt>wmf-config/ConfigSettings.php</tt> changes to switch the datacenter in MediaWiki
#* Flip <tt>appservers-rw, api-rw, imagescaler-rw</tt> to <tt>pooled=false</tt> in the old site. After this, DNS will be changed and internal applications will start hitting the new DC


==== Phase 6 - Redis replicas ====
# Invert the Redis replication for the <code>sessions</code> cluster: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/05-invert-redis-sessions.py|05-invert-redis-sessions.py]]
# Switch the live redis configuration. Verify redises are indeed replicating correctly: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t06_redis.py switchdc t06_redis] [[Switch Datacenter/MediaWiki#output 7|sample output]]


==== Phase 7 - Set new site's databases to read-write ====
==== Phase 6 - Set new site's databases to read-write ====
# Set new-site's core DB masters (shards: s1-s7, x1, es2-es3) in read-write mode: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t07_coredb_masters_readwrite.py switchdc t07_coredb_masters_readwrite.py] [[Switch Datacenter/MediaWiki#output 8|sample output]]
# Set new-site's core DB masters (shards: s1-s8, x1, es4-es5) in read-write mode: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/06-set-db-readwrite.py|06-set-db-readwrite.py]]


==== Phase 8 - Set MediaWiki to read-write ====
==== Phase 7 - Set MediaWiki to read-write ====
# Deploy mediawiki-config <tt>wmf-config/db-$new-site.php</tt> with all shards set to read-write: switchdc [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t08_stop_mediawiki_readonly.py t08_stop_mediawiki_readonly.py] [[Switch Datacenter/MediaWiki#output 9|sample output]]
# Go to read-write mode by changing the <code>ReadOnly</code> conftool value: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/07-set-readwrite.py|07-set-readwrite.py]]


==== Phase 9 - post read-only ====
==== Phase 8 - post read-only ====
# Start maintenance in the new DC: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t09_start_maintenance.py switchdc t09_start_maintenance.py] [[Switch Datacenter/MediaWiki#t09 start maintenance|sample output]]
## the jobqueue in the ''new site'' by running puppet there (mw_primary controls it)
## Run puppet on the maintenance hosts (mw_primary controls it)
# Update tendril for new database masters: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t09_tendril.py switchdc t09_tendril.py] [[Switch Datacenter/MediaWiki#t09 tendril|sample output]]
# Restart parsoid: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t09_restart_parsoid.py switchdc t09_restart_parsoid.py] [[Switch Datacenter/MediaWiki#t09 restart parsoid|sample output]]
# Set the TTL for the DNS records to 300 seconds again: [https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t09_restore_ttl.py switchdc t09_restore_ttl.py] [[Switch Datacenter/MediaWiki#t09 restore ttl|sample output]]
# Update DNS records for new database masters. '''This is not covered by the switchdc script'''
# Start <code>swiftrepl</code> on codfw  '''This is not covered by the switchdc script'''
# [Optional] Run the script to fix broken wikidata entities on the maintenance host of the active datacenter: <code>sudo -u www-data mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki --force</code> '''This is not covered by the switchdc script'''


==== Phase 10 - verification and troubleshooting ====
# Restart Envoy on the jobrunners that are now inactive, to trigger changeprop to re-resolve the DNS name and connect to the new DC: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/08-restart-envoy-on-jobrunners.py|08-restart-envoy-on-jobrunners.py]]
#*A steady rate of 500s is expected until this step is completed, because changeprop will still be sending edits to jobrunners in the old DC, where the database master will reject them.
# Start maintenance in the new DC: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/08-start-maintenance.py|08-start-maintenance.py]]
#* Run puppet on the [[Maintenance_server|maintenance hosts]], which will reactivate systemd timers in both datacenters and add cronjobs in the primary DC
#*Most Wikidata-editing bots will restart once this is done and the "[https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts?orgId=1&refresh=5s dispatch lag]" has recovered. This should bring us back to 100% of editing traffic.
# Update tendril for new database masters: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/08-update-tendril.py|08-update-tendril.py]]
#*Pure cosmetic change, no effect on production. No changes required for database zarcillo (which has a different master for eqiad and codfw).
#*The parsercache hosts and x2 will need to manually be updated in tendril see [[phab:T266723|T266723]]. '''This is not covered by the switchdc script.'''
# Set the TTL for the DNS records to 300 seconds again: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/08-restore-ttl.py|08-restore-ttl.py]]
# Update DNS records for new database masters deploying [https://gerrit.wikimedia.org/r/#/c/operations/dns/+/458787/ eqiad->codfw]; [https://gerrit.wikimedia.org/r/#/c/operations/dns/+/458790/ codfw->eqiad] '''This is not covered by the switchdc script'''. Please use the following to SAL log <code>!log Phase 8.5 Update DNS records for new database masters</code>
# Run Puppet on the database masters in both DCs, to update expected read-only state: [[gerrit:plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/08-run-puppet-on-db-masters.py|08-run-puppet-on-db-masters.py]].
# Make sure the CentralNotice banner informing users of readonly is removed. Keep in mind, there is some minor HTTP caching involved (~5mins)
# Remove the downtime added in phase 0.
#Update [[git:operations/puppet/+/refs/heads/production/modules/profile/files/configmaster/disc desired state.py|disc_desired_state.py]] to reflect which services are pooled in which DCs. See [[phab:T286231|T286231]] as an example. '''This is not covered by the switchdc script.'''
 
==== Phase 9 - verification and troubleshooting ====
'''This is not covered by the switchdc script'''
# Make sure reading & editing works! :)
# Make sure reading & editing works! :)
# Make sure recent changes are flowing (see Special:RecentChanges, [[EventStreams]], [[RCStream]] and the IRC feeds)
# Make sure recent changes are flowing (see Special:RecentChanges, [[EventStreams]], and the IRC feeds)
# Make sure email works (<code>exim4 -bp</code> on mx1001/mx2001, test an email)
# Make sure email works (<code>exim4 -bp</code> on mx1001/mx2001, test an email)
==== Dashboards ====
*[https://grafana.wikimedia.org/d/000000327/apache-fcgi?orgId=1 Apache/FCGI]
*[https://grafana.wikimedia.org/dashboard/db/mediawiki-application-servers?orgId=1 App servers]
*[https://grafana.wikimedia.org/d/kHk7W6OZz/ats-cluster-view?orgId=1&var-datasource=esams%20prometheus%2Fops&var-layer=backend&var-cluster=text ATS cluster view (text)]
*[https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&var-datasource=esams%20prometheus%2Fops&var-cluster=text&var-origin=api-rw.discovery.wmnet&var-origin=appservers-rw.discovery.wmnet&var-origin=restbase.discovery.wmnet ATS backends<->Origin servers overview (appservers, api, restbase)]
*[https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors MediaWiki errors]
*[https://logstash.wikimedia.org/app/kibana#/dashboard/87348b60-90dd-11e8-8687-73968bebd217 Database Errors logs]


=== Media storage/Swift ===
=== Media storage/Swift ===
==== Switchover ====
==== Switchover ====
{{Outdated-inline|note=swiftrepl will now only run in the active datacenter, automatically: [[phab:T285373|T285373]]. (This section will be removed post-switchover)}}
Cache-wise, Swift is active-active. However the background synchronization <code>swiftrepl</code> needs to run in what Mediawiki considers the primary datacenter. A patch such as {{Gerrit|622522}} accomplishes this and needs to be merged after Mediawiki has been switched over (up to e.g. 24/48 hours later is acceptable)


* Set temporary active/active for Swift
==== Switching back ====
# gerrit: C+2 and Submit commit https://gerrit.wikimedia.org/r/#/c/347859/
# <any puppetmaster>: <code>puppet-merge</code>
# <any cumin master>: <code>sudo cumin 'R:class = role::cache::upload and ( *.eqiad.wmnet or *.codfw.wmnet )' 'run-puppet-agent'</code>
* The above must complete correctly and fully (applying the change)
* Set Swift to active/passive in codfw only:
# gerrit: C+2 and Submit commit https://gerrit.wikimedia.org/r/#/c/347860/
# <any puppetmaster>: <code>puppet-merge</code>
# <any cumin master>: <code>sudo cumin 'R:class = role::cache::upload and ( *.eqiad.wmnet or *.codfw.wmnet )' 'run-puppet-agent'</code>


==== Switching back ====
Revert {{Gerrit|622522}} to move <code>swiftrepl</code> back to eqiad


Repeat the steps above in reverse order, with suitable revert commits
==== Dashboards ====
* [https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?orgId=1&var-DC=eqiad&var-prometheus=eqiad%20prometheus%2Fops Swift eqiad]
* [https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?orgId=1&var-DC=codfw&var-prometheus=codfw%20prometheus%2Fops Swift codfw]
* [https://grafana.wikimedia.org/dashboard/db/thumbor?orgId=1 Thumbor]
* [https://grafana.wikimedia.org/d/kHk7W6OZz/ats-cluster-view?orgId=1&var-datasource=esams%20prometheus%2Fops&var-layer=backend&var-cluster=upload ATS cluster view (upload)]
* [https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&var-datasource=esams%20prometheus%2Fops&var-cluster=upload&var-origin=swift.discovery.wmnet ATS backends<->Origin servers overview (swift)]


=== ElasticSearch ===
=== ElasticSearch ===
==== General context on how to switchover ====


CirrusSearch talks by default to the local datacenter ($wmfDatacenter). If Mediawiki switches datacenter, elasticsearch will automatically follow.
CirrusSearch talks by default to the local datacenter (<code>$wmfDatacenter</code>). If Mediawiki switches datacenter, elasticsearch will automatically follow.


Manually switching CirrusSearch to a specific datacenter can always be done. Point CirrusSearch to codfw by editing wmgCirrusSearchDefaultCluster [https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L16025-L16027 InitialiseSettings.php].
Manually switching CirrusSearch to a specific datacenter can always be done. Point CirrusSearch to codfw by editing <code>wmgCirrusSearchDefaultCluster</code> [https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L16025-L16027 InitialiseSettings.php].


To ensure coherence in case of lost updates, a reindex of the pages modified during the switch can be done by following [[Search#Recovering_from_an_Elasticsearch_outage.2Finterruption_in_updates|Recovering from an Elasticsearch outage / interruption in updates]].
To ensure coherence in case of lost updates, a reindex of the pages modified during the switch can be done by following [[Search#Recovering_from_an_Elasticsearch_outage.2Finterruption_in_updates|Recovering from an Elasticsearch outage / interruption in updates]].


=== Traffic ===
==== Preserving more_like query cache performance ====
 
CirrusSearch has a caching layer that caches the result of Elasticsearch queries such as "more like this" queries (which are used, among other things, to generate "Related Articles" at the bottom of mobile Wikipedia pages).
 
Switching datacenters will result in degraded performance while the cache fills back up.
 
In order to avoid the aforementioned performance degradation, a mitigation should be deployed that will hardcode <code>more_like</code> queries to keep routing to the "old" datacenter for 24 hours following the switchover.
 
Hardcoding the cirrus cluster will allow the stampede of cache misses to be sent to the secondary search cluster which has enough capacity, once typical traffic has migrated to the new datacenter, to serve the increased load.
 
This hardcoding should be deployed in advance of the switchover. Since it is effectively a no-op until the actual cutover, it can be deployed as far in advance as desired.
 
For example, if we are switching over from eqiad to codfw, <code>more_like</code> queries should be hardcoded to route to eqiad; this change should be deployed before the actual cutover.
 
Then, 24 hours after the cutover, the hardcoding can be removed, allowing <code>more_like queries</code> to route to the ''new'' cirrus dc - in this example, codfw.


==== General information on generic procedures ====
===== Days in advance preparation =====


https://wikitech.wikimedia.org/wiki/Global_Traffic_Routing
Deploy a patch to hardcode more_like query routing to the currently active DC (i.e. the datacenter we are switching over ''from'').


==== Switchover ====
''Example Patch:'' [mediawiki-config] {{gerrit|635411}} cirrus: Temporarily hardcode more_like query routing


Inter-Cache Routing:
This mitigation should be left in place for 24 hours following the switchover (equivalent to the cache length), at which point there is no longer any performance penalty to removing the hardcoding.
# gerrit: C+2 and Submit commit https://gerrit.wikimedia.org/r/#/c/347613/
# <any puppetmaster>: <code>puppet-merge</code>
# <any cumin master>: <code>sudo cumin 'cp3*.esams.wmnet' 'run-puppet-agent -q'</code>


GeoDNS (User-facing) Routing:
===== One day after datacenter switch =====


# gerrit: C+2 and Submit commit https://gerrit.wikimedia.org/r/#/c/347616
Revert the earlier patch to hardcode more_like query routing; this will allow these queries to route to the newly active DC, and there will not be any performance degradation since the caches have been fully populated by this point.
# <any authdns node>: authdns-update


==== Switchback ====
==== Dashboards ====
* [https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles-prometheus?orgId=1 ElasticSearch Percentiles]


Same procedures as above, with reversions of the commits specified. The switchback will happen in two stages over two days: first reverting the inter-cache routing, then the user routing, to minimize cold-cache issues.
=== Traffic ===
It is relatively straightforward for us to depool an entire datacenter at the traffic level, and is regularly done during maintenance or outages. For that reason, we tend to only keep the datacenter depooled for about a week, which allows us to test for a full traffic cycle (in theory).  


=== Traffic - Services ===
==== General information on generic procedures ====


For reference, the public-facing services involved which are confirmed active/active or failover-capable (other than MW and Swift, handled elsewhere):
See [[Global traffic routing]].
* cache_text:
** restbase (active/passive for now)
** cxserver (active/active)
** citoid (active/active)
* cache_maps:
** kartotherian (active/passive for now)
* cache_misc:
** noc (active/active)
** pybal_config (active/active)
** wdqs (active/active)


==== Switchover ====
==== Switchover ====


* Set temporary active/active for active/passive services above:
GeoDNS (User-facing) Routing:
# gerrit: C+2 and Submit commit https://gerrit.wikimedia.org/r/#/c/347852/
 
# <any puppetmaster>: <code>puppet-merge</code>
# gerrit: C+2 and Submit commit https://gerrit.wikimedia.org/r/#/c/operations/dns/+/458806
# <any cumin master>: <code>sudo cumin '( cp1*.eqiad.wmnet or cp2*.codfw.wmnet ) and R:class ~ "(?i)role::cache::(maps|text)"' 'run-puppet-agent'</code>
# <any authdns node>: authdns-update
* The above must complete correctly and fully (applying the change)
# SAL Log using the following <code>!log Traffic: depool eqiad from user traffic</code>
* Set all active/active (including temps above) to active/passive in codfw only:
# gerrit: C+2 and Submit commit https://gerrit.wikimedia.org/r/#/c/347853/
# <any puppetmaster>: <code>puppet-merge</code>
# <any cumin master>: <code>sudo cumin '( cp1*.eqiad.wmnet or cp2*.codfw.wmnet ) and R:class ~ "(?i)role::cache::(misc|maps|text)"' 'run-puppet-agent'</code>


==== Switchback ====
==== Switchback ====


Reverse the above with reverted commits
Same procedure as above, with reversion of the commit specified: [https://gerrit.wikimedia.org/r/#/c/operations/dns/+/458807 GeoDNS].
 
==== Dashboards ====
*[https://grafana.wikimedia.org/d/000000343/load-balancers-lvs?orgId=1 Load Balancers]
*[https://grafana.wikimedia.org/d/000000479/frontend-traffic?refresh=1m&orgId=1&var-site=codfw&var-site=eqiad&var-cache_type=text&var-status_type=1&var-status_type=2&var-status_type=3&var-status_type=4 Frontend traffic (text@eqiad/codfw)]
*[https://grafana.wikimedia.org/d/000000479/frontend-traffic?refresh=1m&orgId=1&var-site=codfw&var-site=eqiad&var-cache_type=upload&var-status_type=1&var-status_type=2&var-status_type=3&var-status_type=4 Frontend traffic (upload@eqiad/codfw)]


=== Services ===
=== Services ===
All services, are active-active in DNS discovery, apart from restbase, that needs special treatment. The procedure to fail over to one site only is the same for every one of them:
All services, are active-active in DNS discovery, apart from restbase, that needs special treatment. The procedure to fail over to one site only is the same for every one of them:
# reduce the TTL of the dns discovery records to 10 seconds
# reduce the TTL of the DNS discovery records to 10 seconds
# depool the datacenter we're moving away from in confctl / discovery
# depool the datacenter we're moving away from in confctl / discovery
# restore the original TTL
# restore the original TTL
* All of the above is done using the <code>sre.switchdc.services</code> cookbooks:
<syntaxhighlight lang="bash">
# Switch the service "parsoid" to codfw-only
$ cookbook sre.switchdc.services --services parsoid -- eqiad codfw
# Switch all active-active services to codfw, excluding parsoid and cxserver
$ cookbook sre.switchdc.services --exclude parsoid cxserver -- eqiad codfw
</syntaxhighlight>


Restbase is a bit of a special case, and needs an additional step, if we're just switching active traffic over and not simulating a complete failover:
Restbase is a bit of a special case, and needs an additional step, if we're just switching active traffic over and not simulating a complete failover:


# pool restbase-async everywhere, then depool restbase-async in the newly active dc, so that async traffic is separated from real-users traffic as much as possible.
# pool restbase-async everywhere, then depool restbase-async in the newly active dc, so that async traffic is separated from real-users traffic as much as possible.
==== Dashboards ====
*[https://grafana.wikimedia.org/dashboard/db/eventstreams?refresh=1m&orgId=1 EventStreams]
*[https://grafana.wikimedia.org/dashboard/db/ores?refresh=1m&orgId=1 ORES]
*[https://grafana.wikimedia.org/dashboard/db/maps-performances?orgId=1 Maps]
*[https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?refresh=1m WDQS]


=== Other miscellaneous ===
=== Other miscellaneous ===
* [[Switch Datacenter/DeploymentServer|Deployment server]]
* [[Switch Datacenter/DeploymentServer|Deployment server]]
* EventLogging
* [[Analytics/Systems/EventLogging|EventLogging]]
* IRC/RCstream/[[EventStreams]]
* [[Irc.wikimedia.org|IRC]], <s>[[RCStream]]</s>, [[EventStreams]]
 
== Upcoming and past switches ==
{{Anchor|Schedule of past switches}}
 
=== 2021 switches ===
{{tracked|T281515}}
 
;Schedule
* Services: [https://zonestamp.toolforge.org/1624888854 Monday, June 28th, 2021 14:00 UTC]
* Traffic: [https://zonestamp.toolforge.org/1624892434 Monday, June 28th, 2021 15:00 UTC]
* MediaWiki: [https://zonestamp.toolforge.org/1624975258 Tuesday, June 29th, 2021 14:00 UTC]
;Reports
* [https://techblog.wikimedia.org/2021/07/23/june-2021-data-center-switchover/ June 2021 Data Center Switchover] on the Wikimedia Tech blog
*[https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/XI57Z6T2DK7IC345VFTENM5RLTQBQDEQ/ Services and Traffic] on wikitech-l
* [https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/ENL3P5SA7RSOHPN4ILMXQ2BGBF5XR776/ MediaWiki] on wikitech-l (1m57s of read-only time)
* [[Incident documentation/2021-06-29 trwikivoyage primary db]]
 
'''Switching back:'''
 
{{Tracked|T287539}}
* Services: Monday, Sept 13th 14:00 UTC
*Traffic: Monday, Sept 13th 15:00 UTC
*MediaWiki: Tuesday, Sept 14th 14:00 UTC
 
=== 2020 switches ===
{{tracked|T243314}}
;Schedule
* Services: Monday, August 31st, 2020 14:00 UTC
* Traffic: Monday, August 31st, 2020 15:00 UTC
* MediaWiki: Tuesday, September 1st, 2020 14:00 UTC
;Reports
* [[Incident documentation/2020-09-01 data-center-switchover]] (2m49s of read-only time)
 
'''Switching back:'''
* Traffic: Thursday, September 17th, 2020 17:00 UTC
* MediaWiki: Tuesday, October 27th, 2020 14:00 UTC
* Services: Wednesday, October 28th, 2020 14:00 UTC
 
=== 2018 switches ===
{{tracked|T199073}}
;Schedule
* Services: Tuesday, September 11th 2018 14:30 UTC
*Media storage/Swift: Tuesday, September 11th 2018 15:00 UTC
*Traffic: Tuesday, September 11th 2018 19:00 UTC
*MediaWiki: Wednesday, September 12th 2018: 14:00 UTC
;Reports
* [https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/LU63AEQYJDWRN4PL6OHFLT5ENMQBVFMW/ Datacenter Switchover recap] (7m34s of read-only time)
 
'''Switching back:'''
;Schedule
* Traffic: Wednesday, October 10th 2018 09:00 UTC
*MediaWiki: Wednesday, October 10th 2018: 14:00 UTC
*Services: Thursday, October 11th 2018 14:30 UTC
*Media storage/Swift: Thursday, October 11th 2018 15:00 UTC
;Reports
* [https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/YSAZEXIMF73OA3OBC4Z4SYJKC6I4EWJH/ Datacenter Switchback recap] (4m41s of read-only time)
 
=== 2017 switches ===
{{tracked|T138810}}
 
;Schedule
* Elasticsearch: elasticsearch is automatically following mediawiki switch
* Services: Tuesday, April 18th 2017 14:30 UTC
* Media storage/Swift: Tuesday, April 18th 2017 15:00 UTC
* Traffic: Tuesday, April 18th 2017 19:00 UTC
* MediaWiki: Wednesday, April 19th 2017 [https://www.timeanddate.com/worldclock/fixedtime.html?iso=20170419T14 14:00 UTC] (user visible, requires read-only mode)
* Deployment server: Wednesday, April 19th 2017 16:00 UTC
;Reports
* [https://blog.wikimedia.org/2017/04/18/codfw-temporary-editing-pause/ Editing pause for failover test] on Wikimedia Blog
*
 
'''Switching back:'''
;Schedule
* Traffic: Pre-switchback in two phases: Mon May 1 and Tue May 2 (to avoid cold-cache issues Weds)
* MediaWiki:  Wednesday, May 3rd 2017 [https://www.timeanddate.com/worldclock/fixedtime.html?iso=20170503T14 14:00 UTC] (user visible, requires read-only mode)
* Elasticsearch: elasticsearch is automatically following mediawiki switch
* Services: Thursday, May 4th 2017 14:30 UTC
* Swift: Thursday, May 4th 2017 15:30 UTC
* Deployment server: Thursday, May 4th 2017 16:00 UTC
;Reports
* [[Incident documentation/2017-05-03 missing index]]
* [[Incident documentation/2017-05-03 x1 outage]]
 
=== 2016 switches ===
;Schedule
* Deployment server: Wednesday, January 20th 2016
* Traffic: Thursday, March 10th 2016
* MediaWiki 5-minute read-only test: Tuesday, March 15th 2016, 07:00 UTC
* Elasticsearch: Thursday, April 7th 2016, 12:00 UTC
* Media storage/Swift: Thursday, April 14th 2016, 17:00 UTC
* Services: Monday, April 18th 2016, 10:00 UTC
* MediaWiki: Tuesday, April 19th 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)
;Reports
* [https://blog.wikimedia.org/2016/04/11/wikimedia-failover-test/ Wikimedia failover test] on Wikimedia Blog
 
'''Switching back:'''
* MediaWiki: Thursday, April 21st 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)
* Services, Elasticsearch, Traffic, Swift, Deployment server: Thursday, April 21st 2016, after the above is done
 
== Monitoring Dashboards ==
Aggregated list of interesting dashboards
 
*[https://grafana.wikimedia.org/dashboard/db/apache-hhvm?orgId=1 Apache/HHVM]
*[https://grafana.wikimedia.org/dashboard/db/mediawiki-application-servers?orgId=1 App servers]
*[https://grafana.wikimedia.org/dashboard/db/load-balancers?orgId=1 Load Balancers]
*[https://grafana.wikimedia.org/dashboard/db/frontend-traffic?refresh=1m&orgId=1 Frontend traffic]
*[https://grafana.wikimedia.org/dashboard/db/prometheus-varnish-dc-stats?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=cache_upload&var-layear-layer=backend Varnish eqiad]
*[https://grafana.wikimedia.org/dashboard/db/prometheus-varnish-dc-stats?orgId=1&var-datasource=codfw%20prometheus%2Fops&var-cluster=cache_upload&var-layer=backend Varnish codfw]
* [https://grafana.wikimedia.org/dashboard/file/swift.json?orgId=1 Swift eqiad]
* [https://grafana.wikimedia.org/dashboard/file/swift.json?orgId=1&var-DC=codfw Swift codfw]

Revision as of 17:21, 3 August 2021

Introduction

A datacenter switchover (from eqiad to codfw, or vice-versa) comprises switching over multiple different components, some of which can happen independently and many of which need to happen in lockstep. This page documents all the steps needed to switch over from a master datacenter to another one, broken up by component. SRE Service Operations maintains the process and software necessary to run the switchover.

Weeks in advance preparation

  • 10 weeks before: Coordinate dates and communication plan with involved groups: Switch_Datacenter/Coordination
  • 3 weeks before: Run a "live test" of the cookbooks by "switching" from the passive DC to the active DC. The --live-test flag will skip actions that could harm the active DC or do them on the passive DC. This will instrument most of the code paths used in the switchover and help identify issues. This process will !log to SAL so you should coordinate with others, but otherwise should be non-disruptive. Due to changes since the last switchover you can expect code changes to become necessary, so take the time and assistance needed into account.

Overall switchover flow

In a controlled switchover we first deactivate services in the primary datacenter and second deactivate caching in the datacenter. The next step is to switch Mediawiki itself. About a week later we activate caching in the datacenter again, as we believe that testing the situation without caching in the datacenter is sufficient.

Typical scheduling looks like:

  • Monday 14:00 UTC Services
  • Monday 15:00 UTC Caching (traffic)
  • Tuesday 14:00 UTC Mediawiki

The following week reactivate caching. 6+ Weeks later switchback Mediawiki

Per-service switchover instructions

MediaWiki

We divide the process in logical phases that should be executed sequentially. Within any phase, top-level tasks can be executed in parallel to each other, while subtasks are to be executed sequentially to each other. The phase number is referred to in the names of the tasks in the operations/cookbooks repository, in the cookbooks/sre/switchdc/mediawiki/ path.


Days in advance preparation

  1. OPTIONAL: SKIP IN AN EMERGENCY: Make sure databases are in a good state. Normally this requires no operation, as the passive datacenter databases are always prepared to receive traffic, so there are no actionables. Some things that #DBAs normally should make sure for the most optimal state possible (sanity checks):
    • There is no ongoing long-running maintenance that affects database availability or lag (schema changes, upgrades, hardware issues, etc.). Depool those servers that are not ready.
    • Replication is flowing from eqiad -> codfw and from codfw -> eqiad (replication is usually stopped in the passive -> active direction to facilitate maintenance)
    • All database servers have its buffer pool filled up. This is taken care automatically with the automatic buffer pool warmup functionality. For sanity checks, some sample load could be sent to the MediaWiki application server to check requests happen as quickly as in the active datacenter.
    • These were the things we prepared/checked for the 2018 switch
  2. Make absolutely sure that parsercache replication is working from the active to the passive DC. This is important
  3. Check appserver weights on servers in the passive DC, make sure that newer hardware is weighted higher (usually 30) and older hardware is less (usually 25)

Phase 0 - preparation

  1. Disable puppet on maintenance hosts in both eqiad and codfw: 00-disable-puppet.py
  2. Reduce the TTL on appservers-ro, appservers-rw, api-ro, api-rw, jobrunner, videoscaler, parsoid-php to 10 seconds: 00-reduce-ttl.py The warmup step (below) typically takes longer than the old TTL (5 minutes) so this step doesn't wait it to expire. Make sure that at least 5 minutes have passed before moving to Phase 1, the cookbook should force you to wait.
  3. Warm up APC running the mediawiki-cache-warmup on the new site clusters. The warmup queries will repeat automatically until the response times stabilize: 00-warmup-caches.py
    • The global "urls-cluster" warmup against the appservers cluster
    • The "urls-server" warmup against all hosts in the appservers cluster.
    • The "urls-server" warmup against all hosts in the api-appservers cluster.
  4. Set downtime for Read only checks on mariadb masters changed on Phase 3 so they don't page. This is not covered by the switchdc script.

Phase 1 - stop maintenance

  1. Stop maintenance jobs in both datacenters and kill all the periodic jobs (systemd timers and cronjobs) on all maintenance host in both datacenters: 01-stop-maintenance.py

Phase 2 - read-only mode

  1. Go to read-only mode by changing the ReadOnly conftool value: 02-set-readonly.py

Phase 3 - lock down database masters

  1. Put old-site core DB masters (shards: s1-s8, x1, es4-es5) in read-only mode and wait for the new site's databases to catch up replication: 03-set-db-readonly.py

Phase 4 - switch active datacenter configuration

  1. Switch the discovery records and MediaWiki active datacenter: 04-switch-mediawiki.py
    • Flip appservers-ro, appservers-rw, api-ro, api-rw, jobrunner, videoscaler, parsoid-php to pooled=true in the new site. Since both sites are now pooled in etcd, this will not actually change the DNS records for the active datacenter.
    • Flip WMFMasterDatacenter from the old site to the new.
    • Flip appservers-ro, appservers-rw, api-ro, api-rw, jobrunner, videoscaler, parsoid-php to pooled=false in the old site. After this, DNS will be changed for the old DC and internal applications (except mediawiki) will start hitting the new DC

Phase 5 - Invert Redis replication for MediaWiki sessions

  1. Invert the Redis replication for the sessions cluster: 05-invert-redis-sessions.py

Phase 6 - Set new site's databases to read-write

  1. Set new-site's core DB masters (shards: s1-s8, x1, es4-es5) in read-write mode: 06-set-db-readwrite.py

Phase 7 - Set MediaWiki to read-write

  1. Go to read-write mode by changing the ReadOnly conftool value: 07-set-readwrite.py

Phase 8 - post read-only

  1. Restart Envoy on the jobrunners that are now inactive, to trigger changeprop to re-resolve the DNS name and connect to the new DC: 08-restart-envoy-on-jobrunners.py
    • A steady rate of 500s is expected until this step is completed, because changeprop will still be sending edits to jobrunners in the old DC, where the database master will reject them.
  2. Start maintenance in the new DC: 08-start-maintenance.py
    • Run puppet on the maintenance hosts, which will reactivate systemd timers in both datacenters and add cronjobs in the primary DC
    • Most Wikidata-editing bots will restart once this is done and the "dispatch lag" has recovered. This should bring us back to 100% of editing traffic.
  3. Update tendril for new database masters: 08-update-tendril.py
    • Pure cosmetic change, no effect on production. No changes required for database zarcillo (which has a different master for eqiad and codfw).
    • The parsercache hosts and x2 will need to manually be updated in tendril see T266723. This is not covered by the switchdc script.
  4. Set the TTL for the DNS records to 300 seconds again: 08-restore-ttl.py
  5. Update DNS records for new database masters deploying eqiad->codfw; codfw->eqiad This is not covered by the switchdc script. Please use the following to SAL log !log Phase 8.5 Update DNS records for new database masters
  6. Run Puppet on the database masters in both DCs, to update expected read-only state: 08-run-puppet-on-db-masters.py.
  7. Make sure the CentralNotice banner informing users of readonly is removed. Keep in mind, there is some minor HTTP caching involved (~5mins)
  8. Remove the downtime added in phase 0.
  9. Update disc_desired_state.py to reflect which services are pooled in which DCs. See T286231 as an example. This is not covered by the switchdc script.

Phase 9 - verification and troubleshooting

This is not covered by the switchdc script

  1. Make sure reading & editing works! :)
  2. Make sure recent changes are flowing (see Special:RecentChanges, EventStreams, and the IRC feeds)
  3. Make sure email works (exim4 -bp on mx1001/mx2001, test an email)

Dashboards

Media storage/Swift

Switchover

Cache-wise, Swift is active-active. However the background synchronization swiftrepl needs to run in what Mediawiki considers the primary datacenter. A patch such as 622522 accomplishes this and needs to be merged after Mediawiki has been switched over (up to e.g. 24/48 hours later is acceptable)

Switching back

Revert 622522 to move swiftrepl back to eqiad

Dashboards

ElasticSearch

General context on how to switchover

CirrusSearch talks by default to the local datacenter ($wmfDatacenter). If Mediawiki switches datacenter, elasticsearch will automatically follow.

Manually switching CirrusSearch to a specific datacenter can always be done. Point CirrusSearch to codfw by editing wmgCirrusSearchDefaultCluster InitialiseSettings.php.

To ensure coherence in case of lost updates, a reindex of the pages modified during the switch can be done by following Recovering from an Elasticsearch outage / interruption in updates.

Preserving more_like query cache performance

CirrusSearch has a caching layer that caches the result of Elasticsearch queries such as "more like this" queries (which are used, among other things, to generate "Related Articles" at the bottom of mobile Wikipedia pages).

Switching datacenters will result in degraded performance while the cache fills back up.

In order to avoid the aforementioned performance degradation, a mitigation should be deployed that will hardcode more_like queries to keep routing to the "old" datacenter for 24 hours following the switchover.

Hardcoding the cirrus cluster will allow the stampede of cache misses to be sent to the secondary search cluster which has enough capacity, once typical traffic has migrated to the new datacenter, to serve the increased load.

This hardcoding should be deployed in advance of the switchover. Since it is effectively a no-op until the actual cutover, it can be deployed as far in advance as desired.

For example, if we are switching over from eqiad to codfw, more_like queries should be hardcoded to route to eqiad; this change should be deployed before the actual cutover.

Then, 24 hours after the cutover, the hardcoding can be removed, allowing more_like queries to route to the new cirrus dc - in this example, codfw.

Days in advance preparation

Deploy a patch to hardcode more_like query routing to the currently active DC (i.e. the datacenter we are switching over from).

Example Patch: [mediawiki-config] 635411 cirrus: Temporarily hardcode more_like query routing

This mitigation should be left in place for 24 hours following the switchover (equivalent to the cache length), at which point there is no longer any performance penalty to removing the hardcoding.

One day after datacenter switch

Revert the earlier patch to hardcode more_like query routing; this will allow these queries to route to the newly active DC, and there will not be any performance degradation since the caches have been fully populated by this point.

Dashboards

Traffic

It is relatively straightforward for us to depool an entire datacenter at the traffic level, and is regularly done during maintenance or outages. For that reason, we tend to only keep the datacenter depooled for about a week, which allows us to test for a full traffic cycle (in theory).

General information on generic procedures

See Global traffic routing.

Switchover

GeoDNS (User-facing) Routing:

  1. gerrit: C+2 and Submit commit https://gerrit.wikimedia.org/r/#/c/operations/dns/+/458806
  2. <any authdns node>: authdns-update
  3. SAL Log using the following !log Traffic: depool eqiad from user traffic

Switchback

Same procedure as above, with reversion of the commit specified: GeoDNS.

Dashboards

Services

All services, are active-active in DNS discovery, apart from restbase, that needs special treatment. The procedure to fail over to one site only is the same for every one of them:

  1. reduce the TTL of the DNS discovery records to 10 seconds
  2. depool the datacenter we're moving away from in confctl / discovery
  3. restore the original TTL
  • All of the above is done using the sre.switchdc.services cookbooks:
# Switch the service "parsoid" to codfw-only
$ cookbook sre.switchdc.services --services parsoid -- eqiad codfw
# Switch all active-active services to codfw, excluding parsoid and cxserver
$ cookbook sre.switchdc.services --exclude parsoid cxserver -- eqiad codfw

Restbase is a bit of a special case, and needs an additional step, if we're just switching active traffic over and not simulating a complete failover:

  1. pool restbase-async everywhere, then depool restbase-async in the newly active dc, so that async traffic is separated from real-users traffic as much as possible.

Dashboards

Other miscellaneous

Upcoming and past switches

2021 switches

Schedule
Reports

Switching back:

  • Services: Monday, Sept 13th 14:00 UTC
  • Traffic: Monday, Sept 13th 15:00 UTC
  • MediaWiki: Tuesday, Sept 14th 14:00 UTC

2020 switches

Schedule
  • Services: Monday, August 31st, 2020 14:00 UTC
  • Traffic: Monday, August 31st, 2020 15:00 UTC
  • MediaWiki: Tuesday, September 1st, 2020 14:00 UTC
Reports

Switching back:

  • Traffic: Thursday, September 17th, 2020 17:00 UTC
  • MediaWiki: Tuesday, October 27th, 2020 14:00 UTC
  • Services: Wednesday, October 28th, 2020 14:00 UTC

2018 switches

Schedule
  • Services: Tuesday, September 11th 2018 14:30 UTC
  • Media storage/Swift: Tuesday, September 11th 2018 15:00 UTC
  • Traffic: Tuesday, September 11th 2018 19:00 UTC
  • MediaWiki: Wednesday, September 12th 2018: 14:00 UTC
Reports

Switching back:

Schedule
  • Traffic: Wednesday, October 10th 2018 09:00 UTC
  • MediaWiki: Wednesday, October 10th 2018: 14:00 UTC
  • Services: Thursday, October 11th 2018 14:30 UTC
  • Media storage/Swift: Thursday, October 11th 2018 15:00 UTC
Reports

2017 switches

Schedule
  • Elasticsearch: elasticsearch is automatically following mediawiki switch
  • Services: Tuesday, April 18th 2017 14:30 UTC
  • Media storage/Swift: Tuesday, April 18th 2017 15:00 UTC
  • Traffic: Tuesday, April 18th 2017 19:00 UTC
  • MediaWiki: Wednesday, April 19th 2017 14:00 UTC (user visible, requires read-only mode)
  • Deployment server: Wednesday, April 19th 2017 16:00 UTC
Reports

Switching back:

Schedule
  • Traffic: Pre-switchback in two phases: Mon May 1 and Tue May 2 (to avoid cold-cache issues Weds)
  • MediaWiki: Wednesday, May 3rd 2017 14:00 UTC (user visible, requires read-only mode)
  • Elasticsearch: elasticsearch is automatically following mediawiki switch
  • Services: Thursday, May 4th 2017 14:30 UTC
  • Swift: Thursday, May 4th 2017 15:30 UTC
  • Deployment server: Thursday, May 4th 2017 16:00 UTC
Reports

2016 switches

Schedule
  • Deployment server: Wednesday, January 20th 2016
  • Traffic: Thursday, March 10th 2016
  • MediaWiki 5-minute read-only test: Tuesday, March 15th 2016, 07:00 UTC
  • Elasticsearch: Thursday, April 7th 2016, 12:00 UTC
  • Media storage/Swift: Thursday, April 14th 2016, 17:00 UTC
  • Services: Monday, April 18th 2016, 10:00 UTC
  • MediaWiki: Tuesday, April 19th 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)
Reports

Switching back:

  • MediaWiki: Thursday, April 21st 2016, 14:00 UTC / 07:00 PDT / 16:00 CEST (requires read-only mode)
  • Services, Elasticsearch, Traffic, Swift, Deployment server: Thursday, April 21st 2016, after the above is done

Monitoring Dashboards

Aggregated list of interesting dashboards