Anycast authoritative DNS

Work in progress.

In order to improve latency and resilience of our authoritative DNS, this setup leverages BGP and anycast.

Tracking task: https://phabricator.wikimedia.org/T98006

Limitation of a non-anycast setup

By definition, GeoDNS can't be used to redirect users to their closest nameserver (NS), like we do for websites.

When asked for a record (eg. fr.wikipedia.org), the .org zone presents all 3 of our NS to the client, to decide which one to use.

Client side implementations not being great ^{[citation needed]}, anycast offloads that decision to BGP.

Configuration

Server side

The server side is a regular internal anycast setup.

modules/profile/manifests/dns/auth.pp and modules/profile/manifests/dns/recursor.pp include ::profile::bird::anycast

hieradata/role/common/dnsbox.yaml and hieradata/role/common/dns/auth.yaml

profile::bird::advertise_vips:
  nsa.wikimedia.org:
    address: 198.35.27.27 # VIP to advertise (limited to a /32)
    check_cmd: '/usr/lib/nagios/plugins/check_dns_query -H 198.35.27.27 -a -l -d www.wikipedia.org -t 1'
    ensure: present
    service_type: authdns

Routers side

Policy to only create (and thus advertise) the /24 anycast prefix if the router learns about it locally.

policy-options {
    policy-statement BGP_from_anycast {
        term BGP_local_anycast {
            from {
                protocol bgp;
                as-path local_anycast;
            }
            then accept;
        }
        then reject;
    }
    as-path local_anycast "^64605$";
}
routing-options {
    aggregate {
      route 198.35.27.0/24 policy BGP_from_anycast;
    }
}

Troubleshooting

Know which server a client is routed to

$ dig +nsid @nsa.wikimedia.org en.wikipedia.org A |grep NSID

Failure tests

Total local AuthDNS failure

1. Stop gdnsd on all ulsfo servers
2. The anycast prefix stops being advertised to the routers
3. The routers don't have any contributing routes to the less specific prefix
4. The routers stop advertising the prefix to their peers
5. Start gdnsd back
6. prefixes are re-advertised

Limitations

• L3 header LB: ICMP packet too big sent by routers along the path will not consistently be router to the correct router
• Non-consistent hashing: if a routing change on the Internet causes a TCP packet to arrive through a different router, the router will not consistently route it to the proper server