DNS

This article is about DNS as used by the Wikimedia infrastructure. For the public Wikimedia DNS service, see Wikimedia DNS.

This page describes Wikimedia's DNS setup. Wikimedia use two separate kinds of DNS servers, authoritative nameservers (that respond to queries from third party nameservers for our domains) and recursive resolvers (that resolve DNS queries when any of our servers need to look up a name). Both of these services run on our DNS hosts: three each in core sites and two each in the edge, for a total of 16 hosts at the time this page was updated and we added the magru site.

Need to make changes to Wikimedia zones? See #How to make DNS changes. Questions? Please reach out to the Traffic team: #wikimedia-traffic on Libera.

Deploying DNS changes

We use the authdns-update script to deploy DNS changes.

To run authdns-update first merge the patch in the operations/dns.git repository. Then you can SSH to any DNS host under the sudo cumin A:dnsbox alias, such as dns1004.wikimedia.org or dns2004.wikimedia.org. Note that you only need to run the command on one DNS box as it will take care of running it everywhere else.

ssh dns1004.wikimedia.org
sudo -i authdns-update

This will deploy your changes to all DNS hosts.

If you observe any errors while running the script, please bring it up immediately in the #wikimedia-sre or #wikimedia-traffic channel so that the Traffic team can look into it.

If you don't want to remember/find out which DNS host to SSH to (as they change during hardware refreshes), you can always SSH to ns[0-2].wikimedia.org directly. Note that if you have StrictHostKeyChecking enabled, you will need to disable that:

ssh -o StrictHostKeyChecking=no ns0.wikimedia.org
sudo -i authdns-update

Change GeoDNS / Depool a Site

Starting August 2024 and going forward, we no longer GeoDNS depool a site using the admin_state file in the operations/dns repository. Instead, the cookbook sre.dns.admin is used. From any cumin host:

 cookbook sre.dns.admin depool eqiad # [depools eqiad for everything]   
 cookbook sre.dns.admin pool magru   # [pools magru for everything]     
 cookbook sre.dns.admin --service upload-addrs -- depool codfw      # [depool codfw for upload-addrs]
 cookbook sre.dns.admin depool esams --service text-addrs text-next # [depool esams for text*]

If you want to see the current sites/services depooled, run:

 cookbook sre.dns.admin show # this does not make any changes

authdns-update does not need to be run for the cookbook above. Regular deployment of DNS changes is not affected and those still continue to need authdns-update to be run.

Emergency Measures

Update DNS if Gerrit is down

Caution: Announce on IRC that you are going to do this so that no one tries to run authdns-update or the Netbox DNS cookbook in the meantime. This is important as it may break even more things.

In case Gerrit is down, you will need to update the zone files on the DNS hosts without a Gerrit patch. You have two ways to do this and we will assume you want to update the A record for gerrit.wikimedia.org as an example.

Automatic

We recommend this approach but this may require some creative use of sed. As an example, to update the Gerrit DNS record from 208.80.154.151 to 208.80.153.104

Try the sed command on a local DNS repository before editing the production host!

Sed helpers

escaping long strings in an emergency can be tricky, you can use these helpers to prepare your sed commands

# IPv6 auto escape
sed -E 's/[[:space:]]+/\\s+/g; s/:/\\:/g'

# IPv4 auto escape
sed -E 's/[[:space:]]+/\\s+/g; s/\./\\./g'

A live example:

# echo "gerrit-spare 300 IN AAAA 2620:0:860:4:208:80:153:104" | sed -E 's/[[:space:]]+/\\s+/g; s/:/\\:/g'
gerrit-spare\s+300\s+IN\s+AAAA\s+2620\:0\:860\:4\:208\:80\:153\:104

# echo 'gerrit 300 IN A 208.80.154.151'| sed -E 's/[[:space:]]+/\\s+/g; s/\./\\./g'
gerrit\s+300\s+IN\s+A\s+208\.80\.154\.151

Swapping IP on production boxes

Steps outline: update a DNS host (canary), verify changes, and then update all other DNS hosts.

SSH to a cumin host and run
- sudo cumin "A:dnsbox"
- Pick any DNS host from the list as a canary host (all DNS hosts are equal in that respect). We will assume you picked dns1004.wikimedia.org.
Run:
- sudo cumin "P{dns1004*}" "sed -Ei 's/gerrit\s+300\s+IN\s+A\s+208\.80\.154\.151/gerrit 300 IN A 208.80.153.104/' /etc/gdnsd/zones/wikimedia.org"
Verify that the sed command was effective:
- sudo cumin "P{dns1004*}" "grep 208.80.153.104 /etc/gdnsd/zones/wikimedia.org"
- This should suggest that gerrit is now at 208.80.153.104.
Issue gdnsd reload and check name resolution:
- sudo cumin "P{dns1004*}" "gdnsdctl reload-zones && sleep 2 && dig @localhost A +short -p5353 gerrit.wikimedia.org"
- This should return the new IP address for Gerrit.
SSH to dns1004.wikimedia.org and ensure everything looks good (gdnsd reload/journal output looks happy).

If everything looks good in the canary host, simply run the above for all other hosts, skipping the canary host. Be sure to temporary disable puppet on A:dnsbox hosts to ensure manual changes won't be overwritten. Also be sure to re-enable puppet when Gerrit is back online and regular DNS changes can be merged and propagated with the usual workflow.

Disable puppet on A:dnsbox
- sudo cumin "A:dnsbox" 'disable-puppet "INSERT REASON HERE"'
Run:
- sudo cumin "A:dnsbox and not P{dns1004*}" "sed -Ei 's/gerrit\s+300\s+IN\s+A\s+208\.80\.154\.151/gerrit 300 IN A 208.80.153.104/' /etc/gdnsd/zones/wikimedia.org"
Issue gdnsd reload:
- sudo cumin "A:dnsbox and not P{dns1004*}" "gdnsdctl reload-zones"
Enable puppet on A:dnsbox
- sudo cumin "A:dnsbox" 'enable-puppet "INSERT REASON HERE"'

Manual

This means that you need SSH into every DNS box and run gdnsdctl reload-zones. This is a slow and error-prone process, but helps avoid tricky sed expressions.

Steps:

Get a list of the active DNS servers from a cumin host: sudo cumin A:dnsbox
Repeat this step for all the DNS servers in the above list:
1. SSH into the DNS server (example: dns1004.wikimedia.org)
2. Edit the relevant zone file in question and the corresponding record: sudo vim /etc/gdnsd/zones/wikimedia.org to edit the A record for Gerrit.
3. Run sudo gdnsdctl reload-zones to reload gdnsd to pick up the zone file changes.
4. Repeat for all servers to ensure all DNS boxes are in sync.

Update DNS if DNS (recursors) itself is down

This one is a bit tricky as there are many possible corner cases. But in case the internal recursors (10.3.0.1) are themselves down, the good thing is that we have a list of the DNS server IPs in hieradata/common.yaml in the Puppet repository (so no DNS lookup is required for them).

Steps:

Get that list under the authdns_servers key.
Then follow the steps above for the manual DNS update, except that you will need to SSH against the IP and not the hostname.

Authoritative nameservers

In Wikimedia's DNS setup, Wikimedia has 3 authoritative public DNS server Hostnames/IPs, which are referenced by the NS records in all of our zones. These service IPs are routed by bird (unicast and anycast) to any of the 16 available target production hosts whose role contains O:dnsbox.

The three nameserver IPs and their current routing are (as of Sep 2025):

ns0.wikimedia.org - 208.80.154.238 (currently hosted on dns100[4-6].wikimedia.org in eqiad)
ns1.wikimedia.org - 208.80.153.231 (currently hosted on dns200[4-6].wikimedia.org in codfw)
ns2.wikimedia.org - 198.35.27.27 (anycasted, announced from all dns hosts)

All 16 servers which run the authdns setup are picked up by the cumin alias A:dnsbox but A:dns-rec and A:dns-auth also point to the same thing and are preserved for muscle memory. If you wish to query the authdns service on the individual underlying hosts (as opposed to querying the 3 official service IPs, which could be routed to any of them), you'll need to query the non-standard port 5353, as the authdns service doesn't listen on the host IP's port 53 (depending on which subset of hosts, either nothing listens there, or a recursive dns service listens there).

A current full list of DNS server IPs is hardcoded in Puppet and referenced below. For the most up to date list, please refer to the authdns_servers key in Puppet hieradata/common.yaml.

authdns_servers:
  'dns1004.wikimedia.org': 208.80.154.6
  'dns1005.wikimedia.org': 208.80.154.153
  'dns1006.wikimedia.org': 208.80.154.77
  'dns2004.wikimedia.org': 208.80.153.48
  'dns2005.wikimedia.org': 208.80.153.74
  'dns2006.wikimedia.org': 208.80.153.107
  'dns3003.wikimedia.org': 185.15.59.34
  'dns3004.wikimedia.org': 185.15.59.2
  'dns4003.wikimedia.org': 198.35.26.7
  'dns4004.wikimedia.org': 198.35.26.8
  'dns5003.wikimedia.org': 103.102.166.10
  'dns5004.wikimedia.org': 103.102.166.8
  'dns6001.wikimedia.org': 185.15.58.5
  'dns6002.wikimedia.org': 185.15.58.37
  'dns7001.wikimedia.org': 195.200.68.4
  'dns7002.wikimedia.org': 195.200.68.37

The servers are running gdnsd with the geoip plugin which is responsible for geographic DNS.

Zonefiles and other configuration are replicated through the use of git fetch and git merge in a set of update scripts. In case of emergency the servers can be synced from any other as well.

All configuration files can be found in /etc/gdnsd/ on all three hosts, with a separate conf file used just for syncing the zonefiles and such, in /etc/wikimedia-authdns.conf.

The main gdns configuration file is /etc/gdnsd/config. It is generated from files in our operations/dns git repo, as are the zone files.

Debugging Public Nameserver IP mapping

All of our authservers support the NSID EDNS option and are configured to report their internal hostnames in response to NSID queries.

The dig or kdig tools can be used both from the public Internet as well as internal hosts to check which internal authdns server is responding to a request to one of the public authdns IPs by using the +nsid option.

Examples (remove the grep part to see the full output):

sukhe@cumin1002:~$ dig +nsid @ns0.wikimedia.org en.wikipedia.org A | grep NSID
; NSID: 64 6e 73 31 30 30 34 2d 61 75 74 68 ("dns1004-auth")
sukhe@cumin1002:~$ dig +nsid @ns1.wikimedia.org some-random-domain.wikipedia.org A | grep NSID
; NSID: 64 6e 73 32 30 30 36 2d 61 75 74 68 ("dns2006-auth")

# Using kdig from an outside host, for which the routers hash ns2 to dns3002 instead:
blblack@dallas:~$ kdig +nsid @ns2.wikimedia.org en.wikipedia.org A|grep NSID
;; NSID: 646E7333303032 "dns3002"

Domain templates

The zone templates are (regular) files in the operations/dns git repo in the templates/ directory.

Each regular file in this directory corresponds to a zone with the same name. Each symbolic link to a regular file in this directory corresponds to a domain alias. So, in this example:

# ls -l templates/mediawiki*
lrwxrwxrwx    1 root     root           13 Jun 19 15:52 templates/mediawiki.com -> mediawiki.org
lrwxrwxrwx    1 root     root           13 Jun 19 15:52 templates/mediawiki.net -> mediawiki.org
-rw-r--r--    1 root     root         1500 Jun 19 15:12 templates/mediawiki.org

...one zone mediawiki.org is listed, with two alias zones, mediawiki.com and mediawiki.net.

The zonefiles generally follow the standard DNS Master File format (aka BIND zone files) specified by section 5 of RFC1035. gdnsd itself and our jinja-based templating add a few other capabilities on top, notably:

Variables and macros

Within the zone template, a few predefined variables and macros can be used, that will be substituted when the actual zonefiles are generated from the template. These include:

$INCLUDE filename origin: Includes another file, which should be located in a subdirectory
@Z: Is replaced by the actual zone name (FQDN) of the zone (works for symlinks, and also include files)
@F: Is replaced by the original origin at the start of the file, same as @Z in the case of the main zonefile, but can be different in an include file
{{ serial_num }}: Causes an SOA serial number to autogenerated in this place approximating a datestamp of last change, derived from git commit history
{{ serial_comment }}: Emits a text suitable only for a comment line, showing the git hash and the first line of the commit message
{{ langlist(...) }}: A list of language subdomain CNAMEs, i.e. a list of all language abbreviations for all languages any Wikimedia project has, generated from helpers/langlist.tmpl.

Other generic jinja2 templating constructs can be used as well, e.g.:

{% for i in range(1, 10) %}
asw{{ i }}-pmtpa	1H	IN A		10.0.1.{{ i }}
{%- endfor %}

Note that if the range is (a,b) then the first entry will be for a but the last entry will be for b-1.

authdns-local-update

/usr/sbin/authdns-local-update is used on any of the servers for pulling in updates from any other (presumably up to date) dns server. It can be used to bring a server back up to date after e.g. downtime or a software install/update. It is also used in this way by puppet during initial setup.

Geographic DNS

Geographic DNS makes sure that clients end up using the Wikimedia cluster closest to them, by varying DNS responses based on the (country of the) resolver IP querying. This is handled by the gdns geoip plugin. The config file is in config-geo in the operations/dns repo. Our geoip setup makes use of /usr/share/GeoIP/GeoIPCity.dat (ipv4) and /usr/share/GeoIP/GeoIPv6.dat (ipv6). These are pulled from the volatile directory on the puppet master which is updated regularly by cron. See the geoip module for more information.

How to make DNS changes

This section briefly explains how to do the most common DNS changes.

Changing records in a zonefile

This is handled via the git repo operations/dns
Edit the template file templates/zonename locally and check into git, and git review (for gerrit review)
Merge your change in gerrit, then login to dns1004.wikimedia.org (or any DNS host in the cumin alias A:dnsbox), and run sudo -i authdns-update. This will pull from operations/dns and generate zonefiles and gnsd configs on each nameserver.

This no longer requires you to forward your own key, the systems are set up with their own trusted keys for the sync.
Please make sure that the update runs correctly on all dns servers, contact other SREs in case this doesn't happen and you don't know why.

Once the script completes, its a good habit to query all three DNS servers to make sure your change has been correctly deployed
- for example:

for i in 0 1 2; do
   ns=ns${i}.wikimedia.org
   echo $ns
   dig +short @${ns} -t srv _etcd-server-ssl._tcp.dse-k8s-etcd.eqiad.wmnet
done

If any auth DNS server failed to response, please contact other SREs and check what's wrong before doing anything else.

Netbox-driven dynamic data

Most of the infrastructure records are dynamically generated via Netbox-driven data. See DNS/Netbox.

Adding a new zone

First, decide if this new zone will use a new, independent zonefile, or will be an alias of another zone
independent zonefile

Create the new zone template in the operations/dns repository as templates/zonename. (Copy an existing, relatively clean zonefile like wiktionary.org to start with).

zone alias

Make a symbolic link templates/aliasname for the alias to the zone being aliased.
git add the file in, commit, and review on gerrit.
Run authdns-update on ns0 (or any nameserver).
Query all three ns servers to verify that your change took correctly.
If any auth DNS server failed to response, restart it with /etc/init.d/pdns restart

Removing a zone

git rm the appropriate file, and merge on gerrit.
Log into NS0 and run authdns-update

Adding a new (language) wiki

Add the language code to templates/helpers/langlist.tmpl in the operations/dns repo and merge the commit in gerrit.
Follow the normal deployment process described above, running sudo -i authdns-update on dns1004.wikimedia.org or any other A:dnsbox host.

If a certain nameserver is unreachable

When a certain nameserver is unreachable, the others can still be updated from any of the other servers, by running authdns-update there. To skip the unreachable server in the update process, use:

# authdns-update -s "server list"

where server list is a space separated list of FQDNs. Do not forget the quotes, the script will only accept one argument behind -s.

query all three ns servers to verify that your change took correctly.
If any auth DNS server failed to response, restart it with /etc/init.d/pdns restart

Linting the zone files

Most scripts here expect to be run from the root directory of a clone of the operations/dns repository.
All of the python scripts require Python 3.5+ (e.g. stretch python3)

To run locally the same checks of CI:

Partial local CI: install the tox python module and run tox -- -n
Complete local CI: install tox python module and the latest WMF debian package of gdnsd and run: tox

For more detailed instructions see the utils/README file inside the repository.

For more details on the WMF-specific DNS zone files consistency check validator script for the internal zone files (wmnet and pointer zones), see the docstring at the top of the file.

Know to which DC a specific IP is redirected

On any A:dnsbox host:

bblack@dns1004:~$ gdnsd_geoip_test
[starts an interactive shell, then input mapname followed by IP like:]
> generic-map 1.2.3.4
generic-map => 1.2.3.4/24 => eqiad, codfw, ulsfo, esams, eqsin

If you just want to do a single lookup you can put the mapname and IP on the commandline too, but if you're doing a bunch, using the interactive shell prevents expensively reloading maps every time

Know which IP the AuthDNS is seeing a query from

For scenarios where the network you're currently connected is not being redirected to the proper DC, while both your IP and your resolver's IP maps to the proper DC.

Lookup reflect.wikimedia.org, it will tell you how the authdns really sees the client or recursor IP.

Recursive Resolvers

pdns_recursor is run on all DNS hosts. You can get a list by using A:dnsbox or A:dns-rec (same thing).

The recursive cache software used here is PowerDNS recursor (aka pdns-recursor). These machines also host a local authdns instance and site NTP servers.

Statistics are available at: Grafana DNS Recursor Stats

Wipe caches

To wipe records from the recursors caches to clear wrong records, negative caches, etc, you can run the sre.dns.wipe-cache cookbook:

$ sudo cookbook sre.dns.wipe-cache -h
usage: cookbook [GLOBAL_ARGS] sre.dns.wipe-cache [-h] [-s {eqiad,codfw,esams,ulsfo,eqsin,drmrs}] domains [domains ...]

Cookbook to wipedns cache entries.

    Perform the actions outlined in the following wiki:
    https://wikitech.wikimedia.org/wiki/DNS#Wipe_caches

    Usage example:
        cookbook sre.dns.wipe-cache puppet.esqin.wmnet
        cookbook sre.dns.wipe-cache --site esqin puppet.esqin.wmnet
        cookbook sre.dns.wipe-cache ulsfo.wmnet$
        cookbook sre.dns.wipe-cache host1001.eqiad.wmnet host2001.codfw.wmnet


positional arguments:
  domains               The DNS domains to wipe from the cache. Multiple domains can be passed. The domains can be
                        suffixed with a ‘$’. to delete the whole tree from the cache.

optional arguments:
  -h, --help            show this help message and exit
  -s {eqiad,codfw,esams,ulsfo,eqsin,drmrs}, --site {eqiad,codfw,esams,ulsfo,eqsin,drmrs}
                        This is used to limit the recursors that will be wiped. Passing eqiad here will only wipe dns
                        entries from the eqiad servers (default: None)

How to manually remove, a record from the DNS resolver caches (flush / wipe / clear dns resolver cache)

If you have just added or updated a DNS record on the authoritative nameservers, it may still be cached on the (unrelated) DNS resolvers used by our servers. To clear a record from the resolver cache, use:

# rec_control wipe-cache record-name

on all the DNS resolvers. This will also clear any negative cache records. If you need to clear a PTR record, be sure to use the actual record name, e.g.

 # rec_control wipe-cache 122.36.64.10.in-addr.arpa.

(with the trailing '.').

All of the recursors can be targeted via this cumin command:

 $ sudo cumin 'A:dns-rec' 'rec_control wipe-cache <FQDN.of.server>'

How to manually remove an entire domain from the DNS resolver caches

In larger migrations it might be useful to be able to wipe the cache for an entire domain. For domains for which we are authoritative the impact on the resolvers is minimal given that they query a local instance of gdnsd. Care should be used before wiping the cache for an external domain, as it might affect the time it will take to repopulate the cache from the external DNS authoritative resolvers.

To wipe an entire domain the $ suffix can be used as in this example:

$ sudo cumin 'A:dns-rec' 'rec_control wipe-cache eqsin.wmnet$'

For reverse records it can be expanded in the reverse way, for example the following command will wipe all reverse IPv4 records of the form 10.64.*.*:

$ sudo cumin 'A:dns-rec' 'rec_control wipe-cache 64.10.in-addr.arpa$'

For more information see the official documentation at: https://doc.powerdns.com/recursor/manpages/rec_control.1.html.

Service Checks

authdns update run

This alert indicates that there is a discrepancy between the state of the local DNS repo on the DNS hosts and the operations/dns.git repository.

To resolve this, you should run authdns-update from any DNS host. See #Deploying DNS changes

gdnsd checkconf

This alert failing indicates that gdnsd failed trying to validate the configuration and/or zone files. To resolve this, look at the journalctl output for gdnsd.service to see the record(s) in question and ping the Traffic team immediately.

DnsboxServiceMismatch

Noticed this alert firing persistently? Depool the host/service in question; see step 1 below.

This alert signals that a DNS service (such as recdns,ntp-[abc],authdns-ns*) is marked pooled in confctl but we are not advertising the relevant service VIP from the DNS host in question.

To resolve this:

Depool the DNS server for all services before proceeding. This is because the services are related to each other so depooling all services is the preferred way to go. (Replace dns7001 with the host in question below.)
1. Example: sudo confctl --reason 'depooling because of error' select 'name=dns7001.wikimedia.org' set/pooled=no
Look at if bird.service or anycast-healthchecker.service are running and then look at the list of advertised VIPs in /etc/bird/anycast-prefixes.conf.
If everything else fails, ping the Traffic or the Netops team.

External links

http://gdnsd.org/

DNS

Contents

Deploying DNS changes

Change GeoDNS / Depool a Site

Emergency Measures

Update DNS if Gerrit is down

Automatic

Sed helpers

Swapping IP on production boxes

Manual

Update DNS if DNS (recursors) itself is down

Authoritative nameservers

Debugging Public Nameserver IP mapping

Domain templates

Variables and macros

authdns-local-update

Geographic DNS

How to make DNS changes

Changing records in a zonefile

Netbox-driven dynamic data

Adding a new zone

Removing a zone

Adding a new (language) wiki

If a certain nameserver is unreachable

Linting the zone files

Know to which DC a specific IP is redirected

Know which IP the AuthDNS is seeing a query from

Recursive Resolvers

Wipe caches

How to manually remove, a record from the DNS resolver caches (flush / wipe / clear dns resolver cache)

How to manually remove an entire domain from the DNS resolver caches

Service Checks

authdns update run

gdnsd checkconf

DnsboxServiceMismatch

See also

External links

Navigation menu

DNS

Deploying DNS changes

Change GeoDNS / Depool a Site

Emergency Measures

Update DNS if Gerrit is down

Automatic

Sed helpers

Swapping IP on production boxes

Manual

Update DNS if DNS (recursors) itself is down

Authoritative nameservers

Debugging Public Nameserver IP mapping

Domain templates

Variables and macros

authdns-local-update

Geographic DNS

How to make DNS changes

Changing records in a zonefile

Netbox-driven dynamic data

Adding a new zone

Removing a zone

Adding a new (language) wiki

If a certain nameserver is unreachable

Linting the zone files

Know to which DC a specific IP is redirected

Know which IP the AuthDNS is seeing a query from

Recursive Resolvers

Wipe caches

How to manually remove, a record from the DNS resolver caches (flush / wipe / clear dns resolver cache)

How to manually remove an entire domain from the DNS resolver caches

Service Checks

authdns update run

gdnsd checkconf

DnsboxServiceMismatch

See also

External links

Navigation menu

Search