You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Anycast"

From Wikitech-static
Jump to navigation Jump to search
imported>Ayounsi
imported>Sukhbir Singh
m (update notes to declare IPv6 support)
(7 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== External ==
== External ==
In discussion: https://phabricator.wikimedia.org/T98006
PoC: [[Anycast_authoritative_DNS]]
 
PoC: [[Wikidough|Wikidough DoH/DoT recursive resolver]]
 
PoC: [[Durum]]


== Internal ==
== Internal ==
Line 31: Line 35:
# Configure the server side (eg. [[gerrit:c/operations/puppet/+/524037|Gerrit CR 524037]])  
# Configure the server side (eg. [[gerrit:c/operations/puppet/+/524037|Gerrit CR 524037]])  
## Add <code>include ::profile::bird::anycast</code> where you see fit (usually to the service's role)
## Add <code>include ::profile::bird::anycast</code> where you see fit (usually to the service's role)
## Configure the VIP and its attributes (usually <code>hieradata/role/common/</code><syntaxhighlight lang="yaml" line="1">
## Configure the VIP and its required attributes (usually <code>hieradata/role/common/</code><syntaxhighlight lang="yaml" line="1">
profile::bird::advertise_vips:
profile::bird::advertise_vips:
   <vip_fqdn>:  # used as identifier
   <vip_fqdn>:  # used as identifier
     address: 10.3.x.x # VIP to advertise
     address: 10.3.x.x # VIP to advertise
     check_cmd: '/bin/true' # Any command to check the healh of the service
     check_cmd: '/bin/true' # Any command to check the health of the service
    service_type: foobar # Can be any string, if underling applications need to distinguish VIPs
    check_fail: 2 # (Optional, default = 1) number of tries before the service is considered down
</syntaxhighlight>Notes:
</syntaxhighlight>Notes:
##*check_cmd is ran once per second from user "bird"
##*check_cmd is ran once per second from user "bird"
##*If any complex commands create a small bash script or use <code>check_cmd: "/bin/sh -c '<commands>'"</code>
##*If any complex commands create a small bash script or use <code>check_cmd: "/bin/sh -c '<commands>'"</code>
##*anycast-healthchecker use the return code of the heath-check script, 0 = good, everything else is considered as a failure
##*anycast-healthchecker use the return code of the heath-check script, 0 = good, everything else is considered as a failure
##*IPv6 is supported but not enabled by default. See the section below on how to enable it.
# Configure the router side:
# Configure the router side:
##<code>set protocols bgp group Anycast4 neighbor <server_IP></code>
##<code>set protocols bgp group Anycast4 neighbor <server_IP></code>
Line 57: Line 64:
   - routerIP
   - routerIP
   - other_router_IP
   - other_router_IP
# Fast failure detection between router and server (Optional, true by default)
profile::bird::bfd: true
# Add MED=50 to prefixes advertised to secondary routers (Optional, false by default)
# Not merged yet, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/598836
profile::bird::deterministic: false


# Usually set per service (role)
# Usually set per service (role)
Line 64: Line 78:
     address: 10.3.x.x # VIP to advertise (required)
     address: 10.3.x.x # VIP to advertise (required)
     check_cmd: '/bin/true' # Any command to check the healh of the service, ran as user "bird" once per second (required)
     check_cmd: '/bin/true' # Any command to check the healh of the service, ran as user "bird" once per second (required)
    service_type: foobar # Can be any string, if underling applications need to distinguish VIPs (required)
    ensure: present # Set to absent to cleanly remove the check (optional, present by default)
   
# IPv6 support (experimental!)
# IPv6 is not enabled by default and needs to be explicitly enabled as the use case is limited and current deployments use IPv4
# To enable IPv6 support, you need to set do_ipv6 to true and then set the relevant IPv6 settings (address_ipv6 and check_cmd_ipv6)
profile::bird::do_ipv6: true
profile::bird::advertise_vips:
  <vip_fqdn>: # Used as identifier
    address: 10.3.x.x # VIP to advertise (required)
    check_cmd: '/bin/true' # Any command to check the healh of the service, ran as user "bird" once per second (required)
    service_type: foobar # Can be any string, if underling applications need to distinguish VIPs (required)
     ensure: present # Set to absent to cleanly remove the check (optional, present by default)
     ensure: present # Set to absent to cleanly remove the check (optional, present by default)
     bfd: true # Fast failure detection between router and server (Optional, true by default)
     address_ipv6: 2620:0:860 # /128 IPv6 VIP to advertise (required if do_ipv6 is set to true)
    check_cmd_ipv6: '/bin/true' # command to check the health of the service, for IPv6
</syntaxhighlight>
</syntaxhighlight>


Line 77: Line 104:
}
}
local-address 208.80.153.193; # Router's loopback
local-address 208.80.153.193; # Router's loopback
damping;
import anycast_import;  # See below
import anycast_import;  # See below
family inet {
family inet {
Line 101: Line 129:
         prefix-list-filter anycast-internal4 longer; # Only accept prefixes in the defined range
         prefix-list-filter anycast-internal4 longer; # Only accept prefixes in the defined range
     }
     }
     then accept;
     then {
        damping default;
        accept;
    }
}
}
then reject;
then reject;
Line 153: Line 184:


Can it be restarted with <code>service bird start</code> ?
Can it be restarted with <code>service bird start</code> ?
==== Anycast healthchecker not running ====
Same as above.
Logs are in <code>/var/log/anycast-healthchecker/anycast-healthchecker.log</code>
Process is <code>sudo service anycast-healthchecker status</code>
=== Limitations ===
=== Limitations ===


Line 162: Line 201:
=== Future evolution ===
=== Future evolution ===


*IPv6 is supported by both Bird and anycast_healthchecker, but not implemented in Puppet (no current need)
*Upgrade anycast_healthchecker to 0.9.0 or more recent (and rollback https://gerrit.wikimedia.org/r/c/operations/puppet/+/520643)
*Upgrade anycast_healthchecker to 0.9.0 or more recent (and rollback https://gerrit.wikimedia.org/r/c/operations/puppet/+/520643)
*Implement BGP graceful shutdown on the server side to drain traffic before depooling
*Implement BGP graceful shutdown on the server side to drain traffic before depooling
*Send anycast_healthchecker logs to central syslog server
*Send anycast_healthchecker logs to central syslog server

Revision as of 15:11, 12 October 2021

External

PoC: Anycast_authoritative_DNS

PoC: Wikidough DoH/DoT recursive resolver

PoC: Durum

Internal

In production

Anycast recursive DNS

Anycast syslog

How?

How does it work?

https://en.wikipedia.org/wiki/Anycast

  • The VIP (virtual IP) is configured on the servers loopback
  • Bird (routing daemon) advertises the VIP to the routers using BGP
  • (optional) A BFD session is established between Bird and the routers to ensure fast failover in case of server or link failure
  • Anycast_healthchecker monitors the local (anycasted) service by querying it every second
  • If a service failure is detected, the VIP stops being advertised to the routers
  • When the service is restored, anycast_healthchecker waits 10s before re-advertising the IP to avoid flaps
  • The bird service is linked (systemd bind) to the anycast_healthchecker service so bird is stopped if anycast_healthchecker is not running/crashed
  • Time between a local service failure and clients to be redirected to a different server (advertising the same VIP) is 1s max
  • All servers advertise the same VIP worldwide, clients will be be routed to the closest (in the BGP definition) server (same DC, then shorter AS path, etc...) but is not based on latency
  • Routers do per flow load balancing (ECMP) between all local (same site) servers. Hashing is done on IP and port (L4)
  • As last hope backup, in case all servers stop advertising the VIP (eg. global missconfiguration), eqiad and codfw routers have less specific (/30) backup static routes pointing to their local servers

How to deploy a new service?

  1. Assign an IP in DNS, from the 10.3.0.0/24 range - (eg. Gerrit CR 524045)
  2. Configure the server side (eg. Gerrit CR 524037)
    1. Add include ::profile::bird::anycast where you see fit (usually to the service's role)
    2. Configure the VIP and its required attributes (usually hieradata/role/common/
      profile::bird::advertise_vips:
        <vip_fqdn>:  # used as identifier
          address: 10.3.x.x # VIP to advertise
          check_cmd: '/bin/true' # Any command to check the health of the service
          service_type: foobar # Can be any string, if underling applications need to distinguish VIPs
          check_fail: 2 # (Optional, default = 1) number of tries before the service is considered down
      
      Notes:
      • check_cmd is ran once per second from user "bird"
      • If any complex commands create a small bash script or use check_cmd: "/bin/sh -c '<commands>'"
      • anycast-healthchecker use the return code of the heath-check script, 0 = good, everything else is considered as a failure
      • IPv6 is supported but not enabled by default. See the section below on how to enable it.
  3. Configure the router side:
    1. set protocols bgp group Anycast4 neighbor <server_IP>
  4. Add monitoring to the VIP, similar to any Icinga checks, but in modules/profile/manifests/bird/anycast_monitoring.pp
  5. (Optional) if deploying a new type of service, ask Netops to add a backup static route

What other configuration bits are relevant?

Hiera keys:

# service to bind bird to. Usually the anycast-healthchecker
# this mean if anycast-healthchecker crashes, Bird will stop as well
# Usually set globally for Bird
profile::bird::bind_service: 'anycast-healthchecker.service'

# Router IPs with which Birds establish BGP sessions
# Usually set per site
profile::bird::neighbors_list:
  - routerIP
  - other_router_IP

# Fast failure detection between router and server (Optional, true by default)
profile::bird::bfd: true

# Add MED=50 to prefixes advertised to secondary routers (Optional, false by default)
# Not merged yet, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/598836
profile::bird::deterministic: false

# Usually set per service (role)
# But can be set for a specific host as well, for example to specifically remove the VIP from a host to be decommissioned.
profile::bird::advertise_vips:
  <vip_fqdn>: # Used as identifier
    address: 10.3.x.x # VIP to advertise (required)
    check_cmd: '/bin/true' # Any command to check the healh of the service, ran as user "bird" once per second (required)
    service_type: foobar # Can be any string, if underling applications need to distinguish VIPs (required)
    ensure: present # Set to absent to cleanly remove the check (optional, present by default)
    
# IPv6 support (experimental!)
# IPv6 is not enabled by default and needs to be explicitly enabled as the use case is limited and current deployments use IPv4
# To enable IPv6 support, you need to set do_ipv6 to true and then set the relevant IPv6 settings (address_ipv6 and check_cmd_ipv6)
profile::bird::do_ipv6: true
profile::bird::advertise_vips:
  <vip_fqdn>: # Used as identifier
    address: 10.3.x.x # VIP to advertise (required)
    check_cmd: '/bin/true' # Any command to check the healh of the service, ran as user "bird" once per second (required)
    service_type: foobar # Can be any string, if underling applications need to distinguish VIPs (required)
    ensure: present # Set to absent to cleanly remove the check (optional, present by default)
    address_ipv6: 2620:0:860 # /128 IPv6 VIP to advertise (required if do_ipv6 is set to true)
    check_cmd_ipv6: '/bin/true' # command to check the health of the service, for IPv6

How are the routers configured?

# show protocols bgp group Anycast4 
type external;
/* T209989 */
multihop {
    ttl 193;
}
local-address 208.80.153.193; # Router's loopback
damping;
import anycast_import;  # See below
family inet {
    unicast {
        prefix-limit {
            maximum 50; # Take the session down if more than 50 prefixes
            teardown;  # learned from the servers (eg. missconfiguration)
        }
    }
}
export NONE;
peer-as 64605;  # Server's ASN
bfd-liveness-detection {
    minimum-interval 300; # Take the session down after 3*300ms failures
}
multipath;  # Enable load balancing (remove for active/passive)
neighbor 208.80.153.111;  # Servers IPs
neighbor 208.80.153.77;   


# show policy-options policy-statement anycast_import 
term anycast4 {
    from {
        prefix-list-filter anycast-internal4 longer; # Only accept prefixes in the defined range
    }
    then {
        damping default;
        accept;
    }
}
then reject;

# show policy-options prefix-list anycast-internal4      
10.3.0.0/24;

# show routing-options static route 10.3.0.0/30 
next-hop 208.80.153.111;
readvertise;
no-resolve;

How to monitor anycast_healthchecker logs?

/var/log/anycast-healthchecker/anycast-healthchecker.log

How to know which routes a router takes to a specific VIP?

Here both next hops (servers) are load balanced, as they are under the same *[BGP] block.

> show route 10.3.0.1
10.3.0.1/32        *[BGP/170] 1w4d 08:54:21, localpref 100, from 208.80.153.77
                      AS path: 64605 I, validation-state: unverified
                      to 208.80.153.77 via ae3.2003
                    > to 208.80.153.111 via ae4.2004

MTR can also be used for less granularity (site). Eg:

bast5001:~$ mtr 10.3.0.1 --report
Start: Fri Apr  5 16:48:21 2019
HOST: bast5001                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae1-510.cr2-eqsin.wikimed  0.0%    10    0.3   0.7   0.2   4.4   1.1
  2.|-- ae0.cr1-eqsin.wikimedia.o  0.0%    10    0.2   1.0   0.2   7.8   2.3
  3.|-- xe-5-1-2.cr1-codfw.wikime  0.0%    10  195.1 195.3 195.1 196.5   0.3
  4.|-- recdns.anycast.wmnet       0.0%    10  195.1 195.1 195.1 195.1   0.0

How to temporarily depool a server

Disable Puppet, stop the bird service.

How to long term depool a server

Several options:

  • Deactivate the neighbor IP on the router side
  • (Cleaner) Add a specific profile::bird::advertise_vips with the same identifier to the server, and check_cmd: /bin/false or ensure: absent

Troubleshooting

Bird daemon not running

This will trigger an automatic de-pool of the faulty server. Unless this is happening to multiple servers it is not an emergency.

Open a netops task and investigate like any daemon issue.

Has it been shutdown on purpose? Did any change have been made recently?

What do the log says (grep for "bird" in /var/log/syslog)?

Can it be restarted with service bird start ?

Anycast healthchecker not running

Same as above.

Logs are in /var/log/anycast-healthchecker/anycast-healthchecker.log

Process is sudo service anycast-healthchecker status

Limitations

  • The server "self-monitor" itself, if it fails in a way where BGP is up, but DNS is unreachable from outside to the VIP (eg. iptables) this will cause an outage
  • By the nature of Anycast, Icinga will only check the health of the VIP closer to it
    • This could be worked around by checking the anycasted service health from various vantage points (eg. bastion hosts)
    • health checks to the servers' real IP still works

Future evolution