Etcd/Main cluster

The main etcd cluster is the Etcd cluster used as a state management system for the WMF production cluster. It is operated by SRE Service Ops under the etcd main cluster SLO .

Usage in production

More and more systems depend on etcd for retrieving state information. All current uses are listed in the table below

software	use	connection	interval	failure mode
pybal/LVS	retrieve LB pools servers lists, weights, state	custom python/twisted, host only	watch	will keep working until restart
varnish/traffic	retrieve list of backend servers; retrieve VCL fragments (requestctl)	confd (watch)	watch	will keep working
gdnsd/auth dns	Write admin state files for discovery.wmnet records	confd	watch	will keep working
scap/deployment	Dsh lists	confd	60 s	will keep working
MediaWiki	fetch some config variables	PHP connection, request at intervals	10 s	PHP-FPM workloads will keep working until restart, Mw-cron jobs will fail to start
Icinga servers	Update a local cache of the last modified index to be used by other checks	cURL	30 s	the checks will use stale data for comparison
Spicerack	distributed locking for cookbook executions	conftool as a Python library	n.a.	cookbooks can be run without acquiring the lock if needed
Zarcillo	reads candidate master host list	Python etcd client	2 m	graceful/keeps working
Conftool2git	combines audit log events and etcd backend state to mirror history to git	conftool as a Python library	n.a.	audit log events will not trigger updates to the git mirror
Requestctl	web and command line interface for managing WAF rules	conftool as a Python library	n.a.	responders are unable to manage / modify WAF rules

In a failure, all systems will become unable to modify any configuration it derives from etcd, but they will keep working. Only a subset of those will survive a service restart though.

Architecture

The main cluster is composed of two separated sub-clusters: the "codfw.wmnet" and "eqiad.wmnet" ones (creatively name after the datacenters they're located in) that are not connected via RAFT consensus, but via replication, so that there is always a master cluster and a replica one.

Consistency

For reads that don't require sub-second consistency cluster-wide, reading from the replica cluster is acceptable. If replication breaks, this will page opsens that will be able to correct the issue quickly enough (worst case scenario, by pointing clients to the master dc), All writes should go to the master datacenter; we ensure that the replica cluster is in read-only mode for remote clients to avoid issues.

Replication

Replication works using etcdmirror - a pretty raw software we wrote internally that allows replicating from one cluster to another mangling key prefixes. This is supposed to offer the functionality that etcdctl mirror-maker provides on etcd 3 to etcd 2 clusters.

Etcdmirror runs from one machine on the replica cluster (see the profile::etcd::replication::active hiera key). It reads the etcd index to replicate from in /__replication/$destination_prefix (or, if $destination_prefix is the root of the replica cluster keyspace / , to /__replication/__ROOT__ ), issues a recursive watch request to the source cluster starting at the recorded index, and then recursively replicates every write that happens under $source_prefix in the source cluster.

As of April 2024 ( T358636 ), we're replicating nearly the entire keyspace (i.e., / to / ), including /conftool ( conftool state) and /spicerack ( spicerack lock state). One notable exception is /spicerack/locks/etcd which contains short-lived python-etcd lock state that isn't meaningful outside of the source cluster, and is thus ignored by replication.

The logs produced by etcdmirror are pretty verbose, detailing each replication event and any errors encountered should anything go wrong.

Recovering from replication failures

In the event that etcdmirror fails (indicated by the EtcdReplicationDown alert), it should be safe to try restarting the systemd unit if logs suggest a transient issue - e.g., connectivity to the source cluster.

However, etcdmirror is very strict when applying operations to the destination cluster and will fail as soon as any inconsistency is found (even just in the original value of a key) or if the lag is large enough that we're losing etcd events (i.e., when the latest event we've been able to replicate falls outside the 1000 event retention window at the source etcd cluster; see the note in this section of the etcd API docs).

In such a case, you will need to do a full reload. To do that, you need to launch etcdmirror with identical arguments to those used by the systemd unit, but adding the --reload switch. There is a shell script available in /usr/local/sbin on the replication host, which does this for you (look for reload-etdmirror ). Once the reload is complete (look for "Starting replication at" in the logs), you can stop your manual invocation of etcdmirror and restart the systemd unit.

Beware: doing so will ERASE ALL DATA on the destination cluster, so do that with extreme caution.

Individual cluster configuration

We decided to proxy external connections to etcd via an nginx proxy that handles TLS and HTTP authentication and should be fully compliant with etcd's own behaviour. The reason for this is that the builtin authentication gives a severe performance hit to etcd, and that our TLS configuration for nginx is much better than what etcd itself offers. It also gives us the ability to switch on/off the read-only status of a cluster by flipping a switch in puppet. I don't know of any way to do this with the standard etcd mechanism without actually removing users and/or roles, a slow process that is hard to automate/puppetize.

Instead, what happens is that on every cluster member we have an etcd instance listening for client connections on https://$fqdn:2379 with no authentication, but inaccessible to external connections (firewall rules). So local clients, such as etcdmirror, can write to it unauthenticated. At the same time, etcd advertises https://$fqdn:4001 as its client URL, which is also where nginx is listening for external connections and enforces authentication as well.

This can be surprising in exceptional cases when you need to directly modify keys in a cluster where nginx is enforcing read-only mode. If you attempt to do so with etcdctl --endpoints https://$fqdn:2379 , it will still use the advertised URLs and writes will be rejected by nginx.

To work around this, you can use curl to issue the equivalent etcd v2 API calls against https://$fqdn:2379. Again, directly modifying keys is an exceptional operation, so consider getting your commands reviewed by a peer.

The TLS certificates used by etcd - i.e., for peer-peer and client-facing connections (with the sole client being the collocated proxy) - and proxy are managed by our cfssl-based PKI (see task T352245 ). The former uses a custom intermediate to enable TLS client auth for peer-peer connections.

Operations

For the most part, you can refer to what is written in Etcd , but there are a few more operations regarding replication that are not covered there.

Primary cluster switchover

This procedure was refreshed in August 2026 as part of task T433554 and subsequently applied in task T435103 . See task T166552 for historical prior art (2017).

This is a rarely used and high-risk procedure. Carefully review each step for accuracy and relevance before proceeding.

Assuming all patches have been prepared ahead of time, the following procedure incurs an etcd read-only period of 15 - 20 minutes . During that time:

• Mutating operations by confctl , dbctl , and requestctl will fail (reads will succeed); note that this includes conftool-sync during puppet-merge runs for patches that update puppet's conftool-data .
• Cookbooks will be unable to acquire or release Spicerack locks, which are also backed by etcd.

Given this disruption to normal maintenance operations, it is important to communicate the planned switchover to SRE in advance, ideally at least 24 hours ahead of the announced maintenance window, and communicate status proactively in IRC while the process is in flight.

The following procedure contains embedded examples from task T433554 , which switched the primary cluster from eqiad to codfw. When moving in the opposite direction, swap the data centers and / or hosts accordingly in each step.

Unless otherwise noted, commands are run on a cluster-management host. The procedure is broken down into three phases, where the Switchover phase is the disruptive one (read-only). The Preparation and Cleanup phases should happen shortly before and shortly, respectively.

Preparation

1. Reduce the TTL for conftool (R/W) etcd client SRV records to 10 seconds ( DNS#Deploying_DNS_changes ).

• Example: 1319194

2. Silence the EtcdReplicationDown alert shortly before work begins.

Switchover

1. Begin read-only in the etcd cluster we are switching from. Disruption begins.

• Example: 1319191 . Deployed with sudo cumin A:conf-eqiad 'run-puppet-agent'

2. Verify read-only

• Attempting to depool a suitable host should fail (e.g., sudo confctl select "name=$FDQN" set/pooled=no ).

3. Disable puppet on the current and new replication hosts.

• Example: sudo cumin 'conf2005.codfw.wmnet,conf1008.eqiad.wmnet' 'disable-puppet "etcd replication switchover"'

4. Merge a puppet patch that disables etcd replication in the cluster we are switching to and enables it in the cluster we are switching from (i.e., affecting the two hosts on which you just disabled puppet).

• This involves setting profile::etcd::replication::active (default false) appropriately on the hosts involved.
• Example: 1319192 . As noted in the commit message, only the host-level hieradata changes are essential to the switch (i.e., the change to hieradata/role/eqiad/configcluster.yaml is a one-off fix).

5. Run puppet-agent on the current replication host in the cluster we are switching to (replication stops).

• Example:
  • (on conf2005.codfw.wmnet) sudo run-puppet-agent -e "etcd replication switchover"
  • (on conf2005.codfw.wmnet) confirm that etcdmirror--eqiad-wmnet.service has terminated (this may take up to 60 seconds)

6. Set the replication index in the cluster we are switching from.

• Use the python script in P95724 , which can be invoked from any member of the cluster we're switching from (e.g., the new replication host). Note that you will be prompted to approve the etcd write.
• Example:
  • (on conf1008.eqiad.wmnet) python3 set_mirror_index.py --protocol https --old-replica conf2005.codfw.wmnet:4001 --new-replica $(hostname -f):2379 --prefix __ROOT__
  • Note: the prefix is __ROOT__ because we are replicating the full keyspace; see #Replication above.

7. Run puppet-agent on the new replication host in the cluster we are switching from (replication starts).

• Example:
  • (on conf1008.eqiad.wmnet) sudo run-puppet-agent -e "etcd replication switchover"
  • (on conf1008.eqiad.wmnet) confirm that etcdmirror--codfw-wmnet.service has started

8. Test replication via a local mutation on any member of the cluster we are switching to.

• Example:
  • (on conf1008.eqiad.wmnet) Monitor replication progress: journalctl -f -u etcdmirror--codfw-wmnet.service
  • (on conf2005.codfw.wmnet or any other codfw cluster member) Write (and then delete) a suitable test key:
    • curl https://$(hostname -f):2379/v2/keys/test -XPUT -d value=""
    • curl https://$(hostname -f):2379/v2/keys/test -XDELETE

9. Switch the conftool (R/W) etcd client SRV records to the cluster we are switching to ( DNS#Deploying_DNS_changes ).

• Example: 1319195

10. Begin read-write in the etcd cluster we are switching to.

• Example: 1319193 . Deployed with sudo cumin A:conf-codfw 'run-puppet-agent'

11. Verify read-write

• Attempting to depool a suitable host should succeed, e.g., sudo confctl select "name=$FQDN" set/pooled=no .
• Remember to repool the host (e.g., sudo confctl select "name=$FQDN" set/pooled=yes ).

12. Restart hiddenparma ( Requestctl ), since it will have cached connections toward the cluster we are switching from, which is now read-only - i.e., subsequent edits will fail.

• (on alert1002.wikimedia.org, alert2002.wikimedia.org) sudo systemctl restart hiddenparma.service

13. Restart additional read-only services that happen to track the conftool-specific SRV records. Since these are read-only, this is far less urgent than anything above (i.e., they will continue to function even if still connected to the cluster we have switched from).

• Conftool2git - sudo systemctl restart conftool2git.service on the profile::conftool2git::active_host .
• MariaDB/Zarcillo - A service running in the aux-k8s-eqiad cluster. Coordinate with Data Persistence (i.e., ask them to restart it).

Cleanup

1. Delete the EtcdReplicationDown silence.

• If the alert is still firing in the cluster we are switching to (i.e., where etcd-mirror was running previously), it means the ops prometheus host(s) in that DC need a puppet-agent run (you can just wait for this to happen organically before deleting the silence).

2. Restore the TTL for conftool (R/W) etcd client SRV records to 5 minutes ( DNS#Deploying_DNS_changes ).

• Example: 1319196

Depool etcd client traffic from a cluster

If you intend to depool the current primary cluster, you first need to perform a switchover as described above.

In what follows, the target cluster refers to the cluster that will be depooled. For a recent example of this procedure in use, see task T430909 . As always, reality may have changed since the time of this writing, so review the procedure closely for steps that seem unclear or potentially out of date before starting, seeking clarification from your colleagues as needed. Finally, please !log extensively as you perform these steps.

Move PyBals

As of mid-2026, PyBal is still used in core DCs. PyBals in a given DC connect to single static etcd cluster node in that same DC (unless depooled, as we're doing here), specified via the profile::pybal::config_host hiera key in puppet.

• Locate the instance(s) of this hiera key in puppet that configure PyBals to use an etcd node in the target cluster. In practice, this will be one of either hieradata/role/codfw/lvs/balancer.yaml or hieradata/role/eqiad/lvs/balancer.yaml depending on target cluster.
• Switch the value of profile::pybal::config_host to an etcd node in the other cluster. Example: 1308115 .
• Coordinate with Traffic : For each of the LVS hosts affected by your change, run puppet-agent and then restart PyBal.
  • See below for details on how to apply this change to a given LVS host. Additional resources can be found in LVS#Configure_the_load_balancers (e.g., using cumin or the sre.loadbalancer.restart-pybal cookbook).
  • Note that you will typically be touching only PyBals in a single (core) DC, including all 4 of the secondary, low-traffic, high-traffic1, and high-traffic2 instances therein.

Point a PyBal host to a different etcd host

This assumes that the relevant changes have been merged on Gerrit and in Puppet.

• Run puppet on the LVS host to move:

$ sudo run-puppet-agent

• Restart PyBal:

$ sudo systemctl restart pybal.service

• Verify the service and check the journal for potential problems:

$ sudo journalctl -ru pybal.service
-- Journal begins at Wed 2026-06-03 04:17:42 UTC, ends at Wed 2026-08-05 15:55:11 UTC. --
Aug 05 15:54:08 lvs1017 pybal[3089303]: [config-etcd] INFO: connected to etcd://conf2006.codfw.wmnet:4001/conftool/v1/pools/eqiad/cache_text/cdn/                     /eqiad/kubernetes-staging/kubemaster/
Aug 05 15:54:08 lvs1017 pybal[3089303]: [config-etcd] INFO: connected to etcd://conf2006.codfw.wmnet:4001/conftool/v1/pools/eqiad/ncredir/nginx/
...

• Check the output of one (or more) service pool(s), to ensure that they are in the expected state(s):

$ curl localhost:9090/pools
textlb6_80
ncredirlb_80
gerritlb_29418
textlb_80
$ curl localhost:9090/pools/textlb6_80
cp1100.eqiad.wmnet:	enabled/up/pooled
cp1102.eqiad.wmnet:	enabled/up/pooled
[...]

Move other clients

Other clients (confd, MediaWiki, etc.) discover etcd cluster nodes via DNS SRV records. We use a simple geo-mapping technique that creates SRV records in the wmnet zone for each core and caching DC, where the constituent hosts of the SRV record point to the nearest core-DC etcd cluster. These per-DC SRV records are then used by the clients located there. For example, although ulsfo contains no etcd cluster (it's a caching DC), clients located there may use the _etcd-client-ssl._tcp.ulsfo.wmnet SRV record, which resolves to the etcd cluster nodes in codfw.

• Change the _etcd._tcp and _etcd-client-ssl._tcp SRV records in the wmnet zone corresponding to DCs mapped to the target etcd cluster. Example: 1308114 .

There is also a single SRV record in the wikimedia.org zone (see templates/wikimedia.org ). If you are depooling the cluster referenced there, you will need to update these records as well.

• Merge and deploy your change using authdns-update . See DNS#Deploying_DNS_changes .
• You should see traffic start to shift away from the target cluster shortly after. See the etcd grafana dashbord . Note that clients shifting at this stage are limited to those that do not cache connections (so, basically just MediaWiki).
• Wait at least 5 minutes for the SRV TTL to pass, or explicitly clear the recursor caches for each of the SRV records updated using the sre.dns.wipe-cache cookbook. After this is complete, we can begin restarting clients that do cache connections.
• Rolling restart confd in all affected DCs. For example, if you are targeting the codfw cluster, and have thus updated the SRV records used by eqsin, codfw, and ulsfo clients:

sudo cumin -b 16 -s 10 -p 95 'P{C:confd} and (A:eqsin or A:codfw or A:ulsfo)' 'systemctl restart confd'

If you have changed the SRV record in the wikimedia.org zone, the clients of which are not limited to a specific subset of DCs, you can use 'P{C:confd} and P{F:domain = wikimedia.org}' to target the affected confd instances that also require restart.

• Note that this uses a completion percentage below 100 in order for the restart not to abort on the first error (i.e., just in case one of the many affected hosts is transiently unreachange). If you do observe any hosts fail, investigate and whack-a-mole them individually as-needed.
• Restart navtiming.service on the webperf host in the affected core DC (e.g., webperf2003 if you are targeting the codfw cluster).
• Coordinate with Traffic : Restart Liberica daemons in all affected DCs (note: it's really just the control-plane we want to restart, but there's not a separate cookbook for that at this time). For example, to restart Liberica in ulsfo:

sudo cookbook sre.loadbalancer.upgrade -t TXXXXXX --seamless --alias liberica-ulsfo --reason 'Clear control-plane connections to etcd' restart

• Once all of these steps are complete, etcd nodes in the target cluster should no longer receive traffic from clients.
  • Check /var/log/nginx/etcd_access.log and confirm that only /metrics collection remains.
  • Check sudo ss -apn | grep :4001 to confirm that no established TCP connections remain (note this will show nginx as the owning process, which terminates TLS on behalf of etcd clients).

Although it will not appear in the nginx access logs, you may see a trickle of etcd requests in the depooled cluster in the grafana dashboard (mainly PUT / DELETE). This is the local etcdmirror instance replicating from the other cluster - see Etcd/Main_cluster#Replication above.

Repooling

Repooling the cluster follows exactly the same procedure, the only difference being that you are now reverting your earlier puppet and DNS patched (i.e., you apply those changes the same exact way).

Reimage cluster

This procedure was refreshed in August 2026 as part of task T428495 .

Be aware that this might not reflect current reality ! Various details might have changed ... actually everything might have changed. So please use this as a starting template and DOUBLE CHECK EVERY STEP BEFORE YOU EVEN START!

Preparation

• Days before : Review the etcd upgrade guide for important points of note relevant to the specific minor version pair you are upgrading from / to given the specific debian releases involved. This is useful for identifying changes to flag default values, etc. that we may need to explicitly set to retain desired behavior.
  • For example, if you are reimaging from bookworm to trixie, you would review the 3.4 to 3.5 guide.
  • In general, you can disregard the Upgrade procedure section (if present), as it's not appropriate for our production configuration.
• Depool etcd client traffic from the cluster to be reimaged, as described above.
• Verify that profile::etcd::v3::cluster_bootstrap is false for all conf hosts in the cluster.
  • If it is not, you will need to set it to false. Note that applying this change will trigger etcd restarts, so it's preferable to stagger the puppet-agent runs across the 3 cluster member hosts.

There is no way to depool Zookeeper client traffic and clients (e.g., Kafka) will continue to use the cluster while reimages are in flight. This is why it's critical to operate on only one host at a time.

Reimage

One host at a time, execute the following reimage procedure. Do not proceed to the next until both etcd and Zookeeper are healthy.

The following primarily makes use of v3 members API etcdctl commands. However, there are some cases where v2 API commands are used, due to complexities around (lack of) gRPC connectivity to all cluster members.

• Determine whether etcd-mirror is running on this host (i.e., has profile::etcd::replication::active: true ). If so, move it to another cluster member - ideally one that has already been reimaged.
  • Prepare a patch that moves profile::etcd::replication::active: true to the host you would like to move to (i.e., defaults to false on the host you are moving from) and update profile::etcd::replication::dst_url to match. Example: 1314016 .
  • Silence the EtcdReplicationDown alert.
  • Stop puppet on both hosts: sudo disable-puppet "etcd-mirror move"
  • Merge and puppet-merge your patch.
  • (replication stops) Run puppet on the host you are moving from ( sudo run-puppet-agent -e "etcd-mirror move" ). Note that it may take up to 60s for etcd-mirror to terminate.
  • (replication starts) Run puppet on the host you are moving to ( sudo run-puppet-agent -e "etcd-mirror move" ).
• (Optional) Before making destructive changes, if the host you are operating on has not been restarted for some time, consider using the sre.hosts.reboot-single cookbook to verify that you can restart it successfully.
• Determine the member ID of the host you are going to reimage. From any host in the cluster:

ETCDCTL_API=3 etcdctl --endpoints https://$(hostname -f):2379 member list

• Remove the member from the cluster.
  • Note that this will cause etcd on the removed member to leave the cluster and terminate ( docs ). You can monitor this process on the host to be removed with, e.g., journalctl -f -u etcd.service .
  • From any host in the cluster:

ETCDCTL_API=3 etcdctl --endpoints https://$(hostname -f):2379 member remove $MEMBER_ID

• Start the reimage, e.g., sudo cookbook sre.hosts.reimage --os bookworm -t TXXXXX $MEMBER_HOST_NAME (note: not FQDN).
  • The cookbook may recommend you run with --move-vlan . Unless this is what you intend to do, which is a more involved process and not described here, do not do this.
• Any time after the reimage has started, re-add the host to the cluster. With $MEMBER_FQDN as the member you are reimaging, from any other host in the cluster:

ETCDCTL_API=3 etcdctl --endpoints https://$(hostname -f):2379 member add $MEMBER_FQDN --peer-urls=https://$MEMBER_FQDN:2380

• Wait for the reimage to complete. Upon the first puppet run, etcd.service should start and join the (existing) cluster.
• Confirm that the etcd member has joined the cluster. From any host:

ETCDCTL_API=3 etcdctl --endpoints https://$(hostname -f):2379 member list
ETCDCTL_API=2 etcdctl -C https://$(hostname -f):2379 cluster-health  # v2 API (needs connectivity to all members)

• Confirm that the Zookeeper member is healthy and has joined the ensemble.
  • From the reimaged host: echo ruok | nc localhost 2181; echo; echo stat | nc localhost 2181 .
  • From the Zookeeper leader: echo mntr | nc localhost 2181 to confirm the number of synced followers (should be 2).
    • Determine the leader by running echo stat | nc localhost 2181 on each member, e.g., with cumin (look for Mode: leader ).
• Confirm that the nginx etcd TLS proxy is healthy. From any production host, hit 4001 with a known-good path: e.g. curl -v https://$MEMBER_FQDN:4001/v2/keys/ .

Cleanup

• Delete the EtcdReplicationDown silence if it's still active (and not alerting).
• Repool etcd client traffic.

See also

• SLO/etcd main cluster