You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

VXLAN-EVPN Network Testing - Sonic on Dell switches

From Wikitech-static
Jump to navigation Jump to search

Test Topology

This test was done on 4 Dell switches. Two S5232F devices acting as 'Spines' as two S5248F's operating as Leaf/edge switches. All are based on the Broadcom Trident 3 ASIC and ran Dell SONiC-OS-3.4.1-Enterprise_Base. We have 1 server connected to leaf1 on port Ethernet 0 and the other server connected to leaf 2 on port Eth1/1.

Physical Tests

Transceiver Compatibility

FS.com 40GBase-PSM4 QSFP+

   dell-spine1# show interface transceiver Ethernet 4
   
   Ethernet4
   ---------------------------------------------------------------------
   Attribute                :  Value/State
   ---------------------------------------------------------------------
   connector-type           :  MPO
   date-code                :  2020-08-13
   form-factor              :  QSFP+
   cable-length(m)          :  0
   display-name             :  QSFP+ 40GBASE-PSM4
   max-module-power(Watts)  :  3.5
   max-port-power(Watts)    :  5
   vendor-oui               :  44-7C-7F
   present                  :  PRESENT
   serial-no                :  F1930364698
   vendor                   :  FS
   vendor-part              :  QSFP-PLR4-40G
   vendor-rev               :  1B

Finisar 10GBase-LR SFP+

   dell-leaf2# show interface transceiver Ethernet 0
   
   Ethernet0
   ---------------------------------------------------------------------
   Attribute                :  Value/State
   ---------------------------------------------------------------------
   connector-type           :  LC
   date-code                :  2014-07-19
   form-factor              :  SFP+
   cable-length(m)          :  0
   display-name             :  SFP+ 10GBASE-LR
   max-module-power(Watts)  :  2
   max-port-power(Watts)    :  2.5
   vendor-oui               :  00-90-65
   present                  :  PRESENT
   serial-no                :  AS32MX4
   vendor                   :  FINISAR CORP.
   vendor-part              :  FTLX1471D3BCL
   vendor-rev               :  A


FS.com 10GBase-LR SFP+

   dell-leaf2# show interface transceiver Ethernet 1
   
   Ethernet1
   ---------------------------------------------------------------------
   Attribute                :  Value/State
   ---------------------------------------------------------------------
   connector-type           :  LC
   date-code                :  2018-05-04
   form-factor              :  SFP+
   cable-length(m)          :  0
   display-name             :  SFP+ 10GBASE-LR
   max-module-power(Watts)  :  2
   max-port-power(Watts)    :  2.5
   vendor-oui               :  00-90-65
   present                  :  PRESENT
   serial-no                :  G1804036447
   vendor                   :  FiberStore
   vendor-part              :  SFP-10GLR-31
   vendor-rev               :  A

Juniper 10G 3m DAC 10GBASE-CR-DAC-3.0M

   dell-leaf2# show interface transceiver Ethernet 2
   
   Ethernet2
   ---------------------------------------------------------------------
   Attribute                :  Value/State
   ---------------------------------------------------------------------
   date-code                :  2011-11-29
   form-factor              :  SFP+
   cable-length(m)          :  3
   display-name             :  SFP+ 10GBASE-CR-DAC-3.0M
   max-module-power(Watts)  :  2
   max-port-power(Watts)    :  2.5
   vendor-oui               :  41-50-48
   present                  :  PRESENT
   serial-no                :  APF11460020W1Y
   vendor                   :  Amphenol
   vendor-part              :  584990002
   vendor-rev               :  A


FS.com 10G 3m DAC

   dell-leaf2# show interface transceiver Ethernet 4
   
   Ethernet4
   ---------------------------------------------------------------------
   Attribute                :  Value/State
   ---------------------------------------------------------------------
   date-code                :  2019-06-12
   form-factor              :  SFP+
   cable-length(m)          :  3
   display-name             :  SFP+ 10GBASE-CR-DAC-3.0M
   max-module-power(Watts)  :  2
   max-port-power(Watts)    :  2.5
   vendor-oui               :  00-00-00
   present                  :  PRESENT
   serial-no                :  G1906269417-1
   vendor                   :  FS
   vendor-part              :  SFPP-PC03
   vendor-rev               :  A

Juniper 40GBase-SR4 QSFP+

   dell-spine1# show interface transceiver Ethernet 12
   
   Ethernet12
   ---------------------------------------------------------------------
   Attribute                :  Value/State
   ---------------------------------------------------------------------
   connector-type           :  MPO
   date-code                :  2013-12-04
   form-factor              :  QSFP+
   cable-length(m)          :  0
   display-name             :  QSFP+ 40GBASE-SR4
   max-module-power(Watts)  :  1.5
   max-port-power(Watts)    :  5
   vendor-oui               :  00-17-6A
   present                  :  PRESENT
   serial-no                :  QD491487
   vendor                   :  AVAGO
   vendor-part              :  AFBR-79EQDZ-JU1
   vendor-rev               :  01

FS.com SFP 1000BASE-T

The switch is able to see the transceiver but no way to test since the switch doesn't allow to set the interface to 1G.

Ethernet32   SFP 1000BASE-T              1.5   | 2.5     Fiberstore     G1804219349      SFP-GB-GE-T    N/A        N/A          Ready

To set the interface to 1G, you need to change the portgroup first from 25G to 10G, then change the port speed to 1G:

sudo config portgroup speed 9 10000
dell-leaf1(conf-if-Ethernet32)# speed 1000
interface Ethernet32
description test_srv1_1GTBase
mtu 9100
speed 1000
fec none
no shutdown
   dell-leaf1# show interface Ethernet 32
        
   Ethernet32 is up, line protocol is up
   Hardware is Eth
   Description: test_srv1_1GTBase
   Mode of IPV4 address assignment: not-set
   Mode of IPV6 address assignment: not-set
   Interface IPv6 oper status: Disabled
   IP MTU 9100 bytes
   LineSpeed 1GB, Auto-negotiation off
   FEC: DISABLED
   Last clearing of "show interface" counters: never
   10 seconds input rate 0 packets/sec, 0 bits/sec, 0 Bytes/sec
   10 seconds output rate 0 packets/sec, 0 bits/sec, 0 Bytes/sec
   Input statistics:
           28 packets, 2072 octets
           28 Multicasts, 0 Broadcasts, 0 Unicasts
           0 error, 28 discarded, 0 Oversize
           0 Packets (128 to 255 Octects)
   Output statistics:
           2379 packets, 620868 octets
           2379 Multicasts, 0 Broadcasts, 0 Unicasts
           0 error, 0 discarded, 0 Oversize


   dell-leaf1# show mac address-table interface Ethernet 32
   -----------------------------------------------------------
   VLAN         MAC-ADDRESS         TYPE         INTERFACE           
   -----------------------------------------------------------
   2004        D0:94:66:86:4D:6C   DYNAMIC       Ethernet32

'Breakout' Transceivers

4x10G QSFP+ in 'Breakout' Mode

The switch is able to see the break out cable on both ends. the version used is PSM4

Ethernet0    QSFP+ 40GBASE-PSM4          3.5   | 5.0     FS             F1930364699      QSFP-PLR4-40G  N/A        N/A          Ready
Ethernet16   SFP+ 10GBASE-LR             2.0   | 2.5     FINISAR CORP.  AP40JAN          FTLX1471D3BCL  N/A        N/A          Ready

Command for break out

sudo config interface breakout Ethernet0 '4x10G'

After running Logic to limit the impact
Final list of ports to be deleted :
 {
    "Ethernet0": "100000"
}
Final list of ports to be added :
 {
    "Ethernet2": "10000",
    "Ethernet3": "10000",
    "Ethernet0": "10000",
    "Ethernet1": "10000"
}

dell-spine1# show interface breakout
-----------------------------------------------
Port  Breakout Mode  Status        Interfaces
-----------------------------------------------
1/1   4x10G          Completed     Eth1/1/1
                                   Eth1/1/2
                                   Eth1/1/3
                                   Eth1/1/4

4x25G QSFP28 in 'Brekaout' Mode

Not sure we really need this, be good to know if it's an option. I believe it can be done with 100GBase-PSM4 -> 4x25GBase-LR4:

https://community.fs.com/blog/brief-introduction-to-qsfp-100g-psm4-optical-transceiver.html

100GBase-SR4 should also work over MMF to 4x25GBase-SR

Or otherwise a DAC/AOC:

https://www.fs.com/de-en/products/116289.html

https://www.fs.com/products/70439.html

Transceiver DOM Supprt

Digital Optical Monitoring is the standard that transceiver modules use to send statistics to the element they are plugged into (i.e. the switch).

We need to validate that we can see this data on the Dell switches, particularly the Rx light level which is important for longer optical links.

SFP+ / SFP28

Works ok:

   dell-leaf1# show interface transceiver dom Ethernet 0
    
   -----------------------------------------------------------------------
   Ethernet0     
   -----------------------------------------------------------------------
       Identifier: SFP
       Vendor Name: FINISAR CORP.
       Vendor Part: FTLX1471D3BCL
       ChannelMonitorValues:
           Rx1Power: -2.0094 dBm
           Tx1Bias: 41.5540 mA
           Tx1Power: -1.5009 dBm
       ChannelThresholdValues:
           RxPowerHighAlarm  : 2.5001 dBm
           RxPowerHighWarning: 2.0000 dBm
           RxPowerLowAlarm   : -20.0000 dBm
           RxPowerLowWarning : -18.0134 dBm
           TxBiasHighAlarm   : 85.0000 mA
           TxBiasHighWarning : 80.0000 mA
           TxBiasLowAlarm    : 15.0000 mA
           TxBiasLowWarning  : 20.0000 mA
           TxPowerHighAlarm  : 2.0000 dBm
           TxPowerHighWarning: 0.9999 dBm
           TxPowerLowAlarm   : -7.9997 dBm
           TxPowerLowWarning : -7.0006 dBm
       ModuleMonitorValues:
           Temperature: 36.6055 C
           Vcc: 3.3757 Volts
       ModuleThresholdValues:
           TempHighAlarm  : 78.0000 C
           TempHighWarning: 73.0000 C
           TempLowAlarm   : -13.0000 C
           TempLowWarning : -8.0000 C
           VccHighAlarm   : 3.7000 Volts
           VccHighWarning : 3.6000 Volts
           VccLowAlarm    : 2.9000 Volts
           VccLowWarning  : 3.0000 Volts

QSFP+ / QSFP28

   dell-spine1# show interface transceiver dom Eth1/1/1
    
   -----------------------------------------------------------------------
   Eth1/1/1      
   -----------------------------------------------------------------------
       Identifier: QSFP+
       Vendor Name: FS
       Vendor Part: QSFP-PLR4-40G
       ChannelMonitorValues:
           Rx1Power: -1.7289 dBm
           Rx2Power: -2.6898 dBm
           Rx3Power: -2.3980 dBm
           Rx4Power: -inf dBm

(note above is in breakout mode with only 3 ends connected, so makes sense Rx4Power is zero).

Functional Tests

L2 Access port

dell-leaf1# show running-configuration interface Ethernet 0
interface Ethernet0
 description test_srv1
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport access Vlan 2004

dell-leaf1# show running-configuration interface Eth1/1
interface Eth1/1
 description test_Srv2
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport access Vlan 2005
interface Vlan2004
description private1-e-codfw
ip vrf forwarding Vrf_codfw
ip anycast-address 10.192.64.254/22
interface Vlan2005
description private1-f-codfw
ip vrf forwarding Vrf_codfw
ip anycast-address 10.192.80.254/22

Test server 1 in vlan 2004

ppaul@srv1:~$ ip -br link show enp59s0f0
enp59s0f0        UP             40:a8:f0:2c:83:10 <BROADCAST,MULTICAST,UP,LOWER_UP>

Mac is learnt on leaf 1 where test srv1 is connected to port 0, and shown in local Vlan table:

   dell-leaf1# show mac address-table Vlan 2004
   -----------------------------------------------------------
   VLAN         MAC-ADDRESS         TYPE         INTERFACE           
   -----------------------------------------------------------
   2004        40:A8:F0:2C:83:10   DYNAMIC       Ethernet0         


As well as being associated with the attached EVPN VNI:

dell-leaf1# show evpn mac vni 102004
Number of MACs (local and remote) known for this VNI: 2
MAC               Type   Intf/Remote VTEP      VLAN  Seq #'s
00:00:00:10:10:10 local  Vlan2004              2004  0/0
40:a8:f0:2c:83:10 local  Ethernet0             2004  0/0

L2 Trunk port

Interface config:

   dell-leaf1# show running-configuration interface Ethernet 0
   !
   interface Ethernet0
    description test_srv1
    mtu 9100
    speed 10000
    fec none
    no shutdown
    switchport trunk allowed Vlan 2004-2005


With the server configured with an 802.1q sub-interface for each Vlan MACs were learnt correctly on the switch:

   dell-leaf1# show mac address-table interface Ethernet 0
   -----------------------------------------------------------
   VLAN         MAC-ADDRESS         TYPE         INTERFACE           
   -----------------------------------------------------------
   2004        40:A8:F0:2C:83:10   DYNAMIC       Ethernet0           
   2005        40:A8:F0:2C:83:10   DYNAMIC       Ethernet0

Routing to end host from Vlan interface in VRF

IPv4

Server connected via L2 access/trunk port. Switch config:

   interface Vlan2004
    description private1-e-codfw
    ip vrf forwarding Vrf_codfw
    ip anycast-address 10.192.64.254/22
    ipv6 anycast-address 2620:0000:0861:011c::254/64
   

10.192.64.10 is the ip address of test server 1 if we want to ping this test server from leaf 1 we get:

dell-leaf1# ping vrf Vrf_codfw 10.192.64.10
ping: Warning: source address might be selected on device other than Vrf_codfw.
PING 10.192.64.10 (10.192.64.10) from 10.192.64.254 Vrf_codfw: 56(84) bytes of data.
64 bytes from 10.192.64.10: icmp_seq=1 ttl=64 time=0.312 ms
64 bytes from 10.192.64.10: icmp_seq=2 ttl=64 time=0.302 ms
^C
--- 10.192.64.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms

IPv6

The server is configured with IPv6 global unicast address 2620:0000:0861:011c::10, which we also get a response from:

   dell-leaf1# ping vrf Vrf_codfw 2620:0000:0861:011c::10
   ping6: Warning: source address might be selected on device other than Vrf_codfw.
   PING 2620:0000:0861:011c::10(2620:0:861:11c::10) from 2620:0:861:11c::254 Vrf_codfw: 56 data bytes
   64 bytes from 2620:0:861:11c::10: icmp_seq=1 ttl=64 time=0.385 ms
   64 bytes from 2620:0:861:11c::10: icmp_seq=2 ttl=64 time=0.305 ms


Routed port in VRF

Config added as follows to switch port:

   interface Ethernet2
    description srv1_nic2
    mtu 9100
    speed 10000
    fec none
    no shutdown
    ip vrf forwarding Vrf_codfw
    ip address 192.0.2.8/31
    ipv6 address 2620:0000:0861:011e::1/64

Server port IPs were set as follows:

   root@srv1:~# ip -br addr show dev enp59s0f1
   enp59s0f1        UP             192.0.2.9/31 2620:0:861:11e::2/64 fe80::42a8:f0ff:fe2c:8314/64 

IPv4

IPv4 pingable both sides:

   dell-leaf1# ping vrf Vrf_codfw -I 192.0.2.8 192.0.2.9
   PING 192.0.2.9 (192.0.2.9) from 192.0.2.8 Vrf_codfw: 56(84) bytes of data.
   64 bytes from 192.0.2.9: icmp_seq=1 ttl=64 time=0.265 ms
   64 bytes from 192.0.2.9: icmp_seq=2 ttl=64 time=0.250 ms
   root@srv1:~# ping 192.0.2.8
   PING 192.0.2.8 (192.0.2.8) 56(84) bytes of data.
   64 bytes from 192.0.2.8: icmp_seq=1 ttl=64 time=0.171 ms
   64 bytes from 192.0.2.8: icmp_seq=2 ttl=64 time=0.185 ms
   

IPv6

Same with IPv6:

   dell-leaf1# ping vrf Vrf_codfw -I 2620:0000:0861:011e::1  2620:0000:0861:011e::2
   PING 2620:0000:0861:011e::2(2620:0:861:11e::2) from 2620:0:861:11e::1 Vrf_codfw: 56 data bytes
   64 bytes from 2620:0:861:11e::2: icmp_seq=1 ttl=64 time=0.312 ms
   64 bytes from 2620:0:861:11e::2: icmp_seq=2 ttl=64 time=0.247 ms
   root@srv1:~# ping 2620:0000:0861:011e::1
   PING 2620:0000:0861:011e::1(2620:0:861:11e::1) 56 data bytes
   64 bytes from 2620:0:861:11e::1: icmp_seq=1 ttl=64 time=0.209 ms
   64 bytes from 2620:0:861:11e::1: icmp_seq=2 ttl=64 time=0.214 ms

Jumbo frames across L2 Vlan

Same Switch

Remote via VXLAN

Jumbo frames L3 routing

Same Switch

Switch port had MTU set to 9100, Vlan interface did not have a specific MTU configured but it defaults to 9100:

interface Ethernet0
 description test_srv1
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport trunk allowed Vlan 2004-2005
dell-leaf1# show running-configuration interface Vlan 2004
!
interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.64.254/22
 ipv6 anycast-address 2620:0000:0861:011c::254/64
dell-leaf1# show interface Vlan 2004
     
Vlan2004 is up, line protocol is up
Description: private1-e-codfw
Mode of IPV4 address assignment: not-set
Mode of IPV6 address assignment: not-set
Interface IPv6 oper status: Disabled
IP MTU 9100 bytes


Server also had MTU set to 9100:

root@srv1:~# ip -4 -d addr show enp59s0f0.2004
6: enp59s0f0.2004@enp59s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
    link/ether 40:a8:f0:2c:83:10 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    vlan protocol 802.1Q id 2004 <REORDER_HDR> numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 10.192.64.10/22 scope global enp59s0f0.2004
       valid_lft forever preferred_lft forever

With this config we can ping the Vlan interface on the switch with a 9000 byte packet and DF bit set:

root@srv1:~# ping -s 9000 -M do 10.192.64.254
PING 10.192.64.254 (10.192.64.254) 9000(9028) bytes of data.
9008 bytes from 10.192.64.254: icmp_seq=1 ttl=64 time=0.244 ms
9008 bytes from 10.192.64.254: icmp_seq=2 ttl=64 time=0.247 ms
9008 bytes from 10.192.64.254: icmp_seq=3 ttl=64 time=0.260 ms
9008 bytes from 10.192.64.254: icmp_seq=4 ttl=64 time=0.265 ms

Remote via VXLAN

Remote MAC Learning on Vlan/VNI

MAC 40:A8:F0:2C:83:10 is being learnt from srv1 connected to leaf1 on Vlan2004.

Logging on to leaf2 we can see it is receiving an EVPN type 2 route corresponding to this MAC:

   dell-leaf2# show bgp l2vpn evpn route detail 
   BGP routing table entry for 10.0.1.24:2004:[2]:[0]:[48]:[40:a8:f0:2c:83:10]:[32]:[10.192.64.10]
   Paths: (2 available, best #1)
     Advertised to non peer-group peers:
     172.16.1.9 172.16.1.13
     Route [2]:[0]:[48]:[40:a8:f0:2c:83:10]:[32]:[10.192.64.10] VNI 102004/404000
     65030 65032
       10.10.10.1 from 172.16.1.9 (10.0.1.13)
         Origin IGP, valid, external, best (Router ID)
         Extended Community: RT:65032:102004 RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
         Last update: Fri Apr  1 11:27:40 2022
     Route [2]:[0]:[48]:[40:a8:f0:2c:83:10]:[32]:[10.192.64.10] VNI 102004/404000
     65030 65032
       10.10.10.1 from 172.16.1.13 (10.0.1.14)
         Origin IGP, valid, external
         Extended Community: RT:65032:102004 RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
         Last update: Fri Apr  1 11:27:40 2022
   

This is processed correctly and added to the local EVPN database:

   dell-leaf2# show evpn mac vni 102004 mac 40:A8:F0:2C:83:10
   MAC: 40:a8:f0:2c:83:10
    Remote VTEP: 10.10.10.1
    Local Seq: 0 Remote Seq: 0
    Kernel Add: Success, Add ReAttempt:0
    Neighbors:
       10.192.64.10 Active
       2620:0:861:11c::10 Active
       fe80::42a8:f0ff:fe2c:8310 Active

The MAC is also visible in the local L2 forwarding table for Vlan 2004, the "interface" it was "learnt" on shows as VxLAN with the IP of the remote VTEP as expected:

   dell-leaf2# show mac address-table Vlan 2004
   -----------------------------------------------------------
   VLAN         MAC-ADDRESS         TYPE         INTERFACE           
   -----------------------------------------------------------
   2004        40:A8:F0:2C:83:10   DYNAMIC       VxLAN DIP: 10.10.10.1

Client to Client L2 Unicast forwarding

Same Switch (Pure L2)

Ethernet0 and Ethernet2 on Leaf1 were configured as trunks allowing the same Vlans:

   interface Ethernet0
    description test_srv1
    mtu 9100
    speed 10000
    fec none
    no shutdown
    switchport trunk allowed Vlan 2004-2005
   interface Ethernet2
    description srv1_nic2
    mtu 9100
    speed 10000
    fec none
    no shutdown
    switchport trunk allowed Vlan 2004-2005

Both of these ports were connected to the same server, so in order to make it work the second sub-interface was added to a different Linux network namespace:

   root@srv1:~# ip -br link show enp59s0f0.2004
   enp59s0f0.2004@enp59s0f0 UP             40:a8:f0:2c:83:10 <BROADCAST,MULTICAST,UP,LOWER_UP>
   root@srv1:~# ip -br addr show enp59s0f0.2004
   enp59s0f0.2004@enp59s0f0 UP             10.192.64.10/24 2620:0:861:11c::10/64 fe80::42a8:f0ff:fe2c:8310/64
   root@srv1:~# ip route get 10.192.64.30
   10.192.64.30 dev enp59s0f0.2004 src 10.192.64.10 uid 0


   root@srv1:~# ip netns exec TESTNS ip -br link show dev enp59s0f1.2004
   enp59s0f1.2004@if5 UP             40:a8:f0:2c:83:14 <BROADCAST,MULTICAST,UP,LOWER_UP>
   root@srv1:~# ip netns exec TESTNS ip -br addr show dev enp59s0f1.2004
   enp59s0f1.2004@if5 UP             10.192.64.30/24 fe80::42a8:f0ff:fe2c:8314/64
   root@srv1:~# ip netns exec TESTNS ip route get 10.192.64.20
   10.192.64.20 dev enp59s0f1.2004 src 10.192.64.30 uid 0


This use of separate namespaces on the system isolates the two ports, so the "default" namespace, with a port connected to Ethernet0, does not know about the interface in the other one, which is connected to Ethernet2. As such when we send frames from the default namespace for the IP configured on the TESTNS port it will send it out to switch port Ethernet 0, and the switch should forward it via Ethernet 2, looping back to the same server. We only had a single server so this simulates having two separate boxes and sending traffic between them. Results were as expected.

Both MACs are properly learnt on the switch as expected:

   dell-leaf1# show mac address-table Vlan 2004
   -----------------------------------------------------------
   VLAN         MAC-ADDRESS         TYPE         INTERFACE           
   -----------------------------------------------------------
   2004        40:A8:F0:2C:83:10   DYNAMIC       Ethernet0           
   2004        40:A8:F0:2C:83:14   DYNAMIC       Ethernet2

ARP works and pings flow successfully via the switch:

   root@srv1:~# ip neigh show 10.192.64.30
   10.192.64.30 dev enp59s0f0.2004 lladdr 40:a8:f0:2c:83:14 STALE
   root@srv1:~# ip netns exec TESTNS ip neigh show 10.192.64.10
   10.192.64.10 dev enp59s0f1.2004 lladdr 40:a8:f0:2c:83:10 STALE
   root@srv1:~# ping 10.192.64.30
   PING 10.192.64.30 (10.192.64.30) 56(84) bytes of data.
   64 bytes from 10.192.64.30: icmp_seq=1 ttl=64 time=0.167 ms
   64 bytes from 10.192.64.30: icmp_seq=2 ttl=64 time=0.136 ms
   
   root@srv1:~# ip netns exec TESTNS ping -c 2 10.192.64.10
   PING 10.192.64.10 (10.192.64.10) 56(84) bytes of data.
   64 bytes from 10.192.64.10: icmp_seq=1 ttl=64 time=0.138 ms
   64 bytes from 10.192.64.10: icmp_seq=2 ttl=64 time=0.135 ms

TCPDUMP shows the traffic coming in externally with MACs as expected:

   root@srv1:~# tcpdump -e -i enp59s0f0.2004 -l -p -nn icmp
   tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
   13:12:24.539625 40:a8:f0:2c:83:14 > 40:a8:f0:2c:83:10, ethertype IPv4 (0x0800), length 98: 10.192.64.30 > 10.192.64.10: ICMP echo request, id 6059, seq 1, length 64
   13:12:24.539655 40:a8:f0:2c:83:10 > 40:a8:f0:2c:83:14, ethertype IPv4 (0x0800), length 98: 10.192.64.10 > 10.192.64.30: ICMP echo reply, id 6059, seq 1, length 64
   13:12:25.545664 40:a8:f0:2c:83:14 > 40:a8:f0:2c:83:10, ethertype IPv4 (0x0800), length 98: 10.192.64.30 > 10.192.64.10: ICMP echo request, id 6059, seq 2, length 64
   13:12:25.545680 40:a8:f0:2c:83:10 > 40:a8:f0:2c:83:14, ethertype IPv4 (0x0800), length 98: 10.192.64.10 > 10.192.64.30: ICMP echo reply, id 6059, seq 2, length 64

Remote Switch (VXLAN tunneled)

Two servers were connected, SRV1 to Leaf1 Ethernet0, and SRV2 to Leaf2 Eth1/1. These ports were configured as follows:

SRV1 was connected to Vlan2005 on Leaf1 with 802.1q trunk encap:

interface Ethernet0
 description test_srv1
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport trunk allowed Vlan 2004-2005
root@srv1:~# ip -br link show enp59s0f0.2005
enp59s0f0.2005@enp59s0f0 UP             40:a8:f0:2c:83:10 <BROADCAST,MULTICAST,UP,LOWER_UP>

The MAC is learnt locally on Leaf1 as expected:

dell-leaf1# show mac address-table interface Ethernet 0 | grep 2005
2005        40:A8:F0:2C:83:10   DYNAMIC       Ethernet0

SRV2 was connected to Vlan2005 on Leaf2 as a regular access port:

interface Eth1/1
 description test_Srv2
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport access Vlan 2005
root@srv2:~# ip -br link show dev enp59s0f0
enp59s0f0        UP             40:a8:f0:2c:31:68 <BROADCAST,MULTICAST,UP,LOWER_UP>

Again the MAC is properly learnt on Leaf2:

dell-leaf2# show mac address-table interface Eth1/1
-----------------------------------------------------------
VLAN         MAC-ADDRESS         TYPE         INTERFACE
-----------------------------------------------------------
2005        40:A8:F0:2C:31:68   DYNAMIC       Eth1/1


In terms of EVPN we can see that Leaf1 receives an both a plain MAC-only EVPN route for this address, as well as one with the IP configured on SRV2 included. Both have a next-hop of Leaf2's Loopback1 interface (VTEP IP):

BGP routing table entry for 10.0.1.25:2005:[2]:[0]:[48]:[40:a8:f0:2c:31:68]
Paths: (2 available, best #2)
  Advertised to non peer-group peers:
  172.16.1.1 172.16.1.5
  Route [2]:[0]:[48]:[40:a8:f0:2c:31:68] VNI 102005
  65030 65033
    10.10.10.2 from 172.16.1.1 (10.0.1.13)
      Origin IGP, valid, external
      Extended Community: RT:65033:102005 ET:8
      Last update: Fri Apr  8 15:05:41 2022
  Route [2]:[0]:[48]:[40:a8:f0:2c:31:68] VNI 102005
  65030 65033
    10.10.10.2 from 172.16.1.5 (10.0.1.14)
      Origin IGP, valid, external, best (Older Path)
      Extended Community: RT:65033:102005 ET:8
      Last update: Tue Apr  5 14:58:20 2022
BGP routing table entry for 10.0.1.25:2005:[2]:[0]:[48]:[40:a8:f0:2c:31:68]:[32]:[10.192.80.10]
Paths: (2 available, best #2)
  Advertised to non peer-group peers:
  172.16.1.1 172.16.1.5
  Route [2]:[0]:[48]:[40:a8:f0:2c:31:68]:[32]:[10.192.80.10] VNI 102005/404000
  65030 65033
    10.10.10.2 from 172.16.1.1 (10.0.1.13)
      Origin IGP, valid, external
      Extended Community: RT:65033:102005 RT:65033:404000 ET:8 Rmac:3c:2c:30:4c:81:83
      Last update: Fri Apr  8 15:05:41 2022
  Route [2]:[0]:[48]:[40:a8:f0:2c:31:68]:[32]:[10.192.80.10] VNI 102005/404000
  65030 65033
    10.10.10.2 from 172.16.1.5 (10.0.1.14)
      Origin IGP, valid, external, best (Older Path)
      Extended Community: RT:65033:102005 RT:65033:404000 ET:8 Rmac:3c:2c:30:4c:81:83
      Last update: Tue Apr  5 14:58:20 2022

This is properly processed and the EVPN database lists it correctly:

dell-leaf1# show evpn mac vni 102005 mac 40:a8:f0:2c:31:68
MAC: 40:a8:f0:2c:31:68
 Remote VTEP: 10.10.10.2
 Local Seq: 0 Remote Seq: 0
 Kernel Add: Success, Add ReAttempt:0
 Neighbors:
    10.192.80.10 Active
    2620:0:861:cabf::10 Active
    fe80::42a8:f0ff:fe2c:3168 Active

When we look at the local Vlan forwarding table on Leaf1 we can see that this MAC has been added, with the remote VTEP listed against it:

dell-leaf1# show mac address-table dynamic Vlan 2005
-----------------------------------------------------------
VLAN         MAC-ADDRESS         TYPE         INTERFACE
-----------------------------------------------------------
2005        40:A8:F0:2C:31:68   DYNAMIC       VxLAN DIP: 10.10.10.2
2005        40:A8:F0:2C:83:10   DYNAMIC       Ethernet0

The reverse is true on Leaf2 for SRV1's MAC learnt on Leaf1, I omit the EVPN details as they work/look much the same in reverse:

dell-leaf2# show mac address-table dynamic Vlan 2005
-----------------------------------------------------------
VLAN         MAC-ADDRESS         TYPE         INTERFACE   
-----------------------------------------------------------
2005        40:A8:F0:2C:31:68   DYNAMIC       Eth1/1  
2005        40:A8:F0:2C:83:10   DYNAMIC       VxLAN DIP: 10.10.10.1

With this in place we should be able to send unicast Ethernet frames between the two servers, which the switches will send over the IP underlay using VXLAN. We test this with a simple ping from SRV1 to SRV2:

root@srv1:~# ping -c 2 10.192.80.10
PING 10.192.80.10 (10.192.80.10) 56(84) bytes of data.
64 bytes from 10.192.80.10: icmp_seq=1 ttl=64 time=0.206 ms
64 bytes from 10.192.80.10: icmp_seq=2 ttl=64 time=0.179 ms
--- 10.192.80.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1029ms
rtt min/avg/max/mdev = 0.179/0.192/0.206/0.019 ms

Doing a TCPdump we can see the MAC addresses are expected:

root@srv2:~# tcpdump -e -i enp59s0f0 -l -p -nn icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:08:46.807595 40:a8:f0:2c:83:10 > 40:a8:f0:2c:31:68, ethertype IPv4 (0x0800), length 98: 10.192.80.187 > 10.192.80.10: ICMP echo request, id 23100, seq 1, length 64
09:08:46.807628 40:a8:f0:2c:31:68 > 40:a8:f0:2c:83:10, ethertype IPv4 (0x0800), length 98: 10.192.80.10 > 10.192.80.187: ICMP echo reply, id 23100, seq 1, length 64
09:08:47.827376 40:a8:f0:2c:83:10 > 40:a8:f0:2c:31:68, ethertype IPv4 (0x0800), length 98: 10.192.80.187 > 10.192.80.10: ICMP echo request, id 23100, seq 2, length 64
09:08:47.827403 40:a8:f0:2c:31:68 > 40:a8:f0:2c:83:10, ethertype IPv4 (0x0800), length 98: 10.192.80.10 > 10.192.80.187: ICMP echo reply, id 23100, seq 2, length 64

Client to Client broadcast forwarding / ingress replication

Same Switch (Pure L2)

Config the same as in 3.8.1

ARP cache was deleted in the TESTNS namespace / on port enp59s0f1, before initiating a ping:

   root@srv1:~# ip netns exec TESTNS bash
   root@srv1:~# ip neigh del 10.192.64.10 dev enp59s0f1.2004
   root@srv1:~# ping 10.192.64.10
   PING 10.192.64.10 (10.192.64.10) 56(84) bytes of data.
   64 bytes from 10.192.64.10: icmp_seq=1 ttl=64 time=0.388 ms
   64 bytes from 10.192.64.10: icmp_seq=2 ttl=64 time=0.127 ms

TCPDUMP in the default netns show's the broadcast arriving on the port from switch:

   root@srv1:~# tcpdump -i enp59s0f0.2004 -e -l -p -nn arp
   tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
   13:30:08.545177 40:a8:f0:2c:83:14 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.192.64.10 tell 10.192.64.30, length 46

Remote Switch (VXLAN tunneled)

Switch and server ports were configured the same as for the unicast test 3.8.2.

We do not have IP Multicast configured in the underlay network, so instead Ingress Replication is used. In this case the edge device receiving the BUM frame duplicates it and sends a copy to each remote switch that has originated an EVPN type 3 route to say it is participating in that VNI/Vlan.

In our lab Leaf1 and Leaf2 both have Vlan2005 configured, bound to VXLAN VNI 102005. We thus see EVPN type 3 routes on each that have originated on the remote side:

Leaf1 output showing route from Leaf2:

dell-leaf1# show bgp l2vpn evpn route detail type multicast | find 10.0.1.25
Route Distinguisher: 10.0.1.25:2005
BGP routing table entry for 10.0.1.25:2005:[3]:[0]:[32]:[10.10.10.2]
Paths: (2 available, best #2)
  Advertised to non peer-group peers:
  172.16.1.1 172.16.1.5
  65030 65033
    10.10.10.2 from 172.16.1.1 (10.0.1.13)
      Origin IGP, valid, external
      Extended Community: RT:65033:102005 ET:8
      Last update: Fri Apr  8 15:05:42 2022
      PMSI Tunnel Type: Ingress Replication, label: 102005
  65030 65033
    10.10.10.2 from 172.16.1.5 (10.0.1.14)
      Origin IGP, valid, external, best (Older Path)
      Extended Community: RT:65033:102005 ET:8
      Last update: Tue Apr  5 14:58:21 2022
      PMSI Tunnel Type: Ingress Replication, label: 102005

This is also visible in the below command, with the loopback1 IP of Leaf2 (10.10.10.2) shown as a remote VTEP with flooding enabled:

dell-leaf1# show evpn vni 102005
VNI: 102005
 Type: L2
 Tenant VRF: default
 Client State: Up
 VxLAN interface: vtep1-2005
 VxLAN ifIndex: 134
 Local VTEP IP: 10.10.10.1
 Mcast group: 0.0.0.0
 Remote VTEPs for this VNI:
  10.10.10.2 flood: HER
  Kernel Add: Success, Add ReAttempt:0
 Number of MACs (local and remote) known for this VNI: 2
 Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 6
 Advertise-gw-macip: No


Leaf2 output shows a similar route, but for Leaf1, and again the EVPN database shows the remote IP as being part of the VNI:

dell-leaf2# show bgp l2vpn evpn route detail type multicast | find 2005
Route Distinguisher: 10.0.1.24:2005
BGP routing table entry for 10.0.1.24:2005:[3]:[0]:[32]:[10.10.10.1]
Paths: (2 available, best #2)
  Advertised to non peer-group peers:
  172.16.1.9 172.16.1.13
  65030 65032
    10.10.10.1 from 172.16.1.9 (10.0.1.13)
      Origin IGP, valid, external
      Extended Community: RT:65032:102005 ET:8
      Last update: Fri Apr  8 15:05:42 2022
      PMSI Tunnel Type: Ingress Replication, label: 102005
  65030 65032
    10.10.10.1 from 172.16.1.13 (10.0.1.14)
      Origin IGP, valid, external, best (Older Path)
      Extended Community: RT:65032:102005 ET:8
      Last update: Tue Apr  5 14:58:21 2022
      PMSI Tunnel Type: Ingress Replication, label: 102005
dell-leaf2# show evpn vni 102005
VNI: 102005
 Type: L2
 Tenant VRF: Vrf_codfw
 Client State: Up
 VxLAN interface: vtep1-2005
 VxLAN ifIndex: 77
 Local VTEP IP: 10.10.10.2
 Mcast group: 0.0.0.0
 Remote VTEPs for this VNI:
  10.10.10.1 flood: HER
  Kernel Add: Success, Add ReAttempt:0
 Number of MACs (local and remote) known for this VNI: 3
 Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 10
 Advertise-gw-macip: No


This should ensure that all broadcast frames transmitted by SRV1 will be received by SRV2 (possibly with the exception of ARP due to ARP suppression).

To test we generate a random broadcast frame on SRV1, connected to Leaf1:

root@srv1:~# echo "test broadcast in vlan2005" | nc -q1 -u -s 10.192.80.187 -b 255.255.255.255 12345

A TCPdump shows this is received as expected on SRV2, connected to Leaf2:

root@srv2:~# tcpdump -X -e -i enp59s0f0 -l -p -nn src 10.192.80.187
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:35:31.670833 40:a8:f0:2c:83:10 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 69: 10.192.80.187.34163 > 255.255.255.255.12345: UDP, length 27
   0x0000:  4500 0037 a5d8 4000 4011 3963 0ac0 50bb  E..7..@.@.9c..P.
   0x0010:  ffff ffff 8573 3039 0023 68d1 7465 7374  .....s09.#h.test
   0x0020:  2062 726f 6164 6361 7374 2069 6e20 766c  .broadcast.in.vl
   0x0030:  616e 3230 3035 0a                        an2005.

We can conclude that ingress replication is thus working as it should for broadcast traffic.

Client to Client L2 multicast forwarding / ingress replication

Same Switch (Pure L2)

Config the same as in 3.8.1.

Multicast generated in TESTNS with Netcat:

   root@srv1:~# echo "test multicast in vlan2004" | nc -q1 -u -b 224.0.0.26 12345

This can be observed leaving the server in a TCPdump:

root@srv1:~# tcpdump -e -i enp59s0f1.2004 -l -p -nn
listening on enp59s0f1.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
17:02:40.891494 40:a8:f0:2c:83:14 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 69: 10.192.64.129.38067 > 224.0.0.26.12345: UDP, length 27

The frame is recieved on the other port having been flooded within the Vlan by the switch:

root@srv1:~# tcpdump -X -e -i enp59s0f0 -l -nn net 224.0.0.0/8
listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:02:40.891570 40:a8:f0:2c:83:14 > 01:00:5e:00:00:1a, ethertype 802.1Q (0x8100), length 73: vlan 2004, p 0, ethertype IPv4, 10.192.64.129.38067 > 224.0.0.26.12345: UDP, length 27
 0x0000:  4500 0037 6f76 4000 0111 dee4 0ac0 4081  E..7ov@.......@.
 0x0010:  e000 001a 94b3 3039 0023 73a4 7465 7374  ......09.#s.test
 0x0020:  206d 756c 7469 6361 7374 2069 6e20 766c  .multicast.in.vl
 0x0030:  616e 3230 3034 0a                        an2004.

Remote Switch (VXLAN tunneled)

Two servers are connected, one to each Leaf, the same as in test 3.8.2 / 3.9.2.

We generate a multicast frame on SRV1:

root@srv1:~# echo "test multicast in vlan2005" | nc -q1 -u -s 10.192.80.187 -b 224.0.0.26 12345 

This can be seen on the wire going out from SRV1 as expected:

root@srv1:~# tcpdump -e -i enp59s0f0.2005 -l -p -nn net 224.0.0.0/8
listening on enp59s0f0.2005, link-type EN10MB (Ethernet), capture size 262144 bytes
18:08:46.017656 40:a8:f0:2c:83:10 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 69: 10.192.80.187.35480 > 224.0.0.26.12345: UDP, length 27

The frame is received on SRV2, connected to the same Vlan but off Leaf2, showing the multicasts have been sent over the VXLAN fabric as required:

root@srv2:~# tcpdump -X -i enp59s0f0 -l -nn -e net 224.0.0.0/8
listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:09:04.782380 40:a8:f0:2c:83:10 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 69: 10.192.80.187.35480 > 224.0.0.26.12345: UDP, length 27
 0x0000:  4500 0037 9a63 4000 0111 a3bd 0ac0 50bb  E..7.c@.......P.
 0x0010:  e000 001a 8a98 3039 0023 6d84 7465 7374  ......09.#m.test
 0x0020:  206d 756c 7469 6361 7374 2069 6e20 766c  .multicast.in.vl
 0x0030:  616e 3230 3035 0a                        an2005.

Inter-Vlan/subnet routing via IRB interfaces on same switch

Leaf1 has Vlan interfaces for Vlan2004 and Vlan2005 configured:

interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 ip anycast-address 10.192.64.254/24
 ipv6 anycast-address 2620:0:861:11c::254/64
interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 ip anycast-address 10.192.80.254/22
 ipv6 anycast-address 2620:0:861:cabf::254/64

IPv4

root@srv1:~# mtr --address 10.192.64.129 -n -r 10.192.80.187
Start: 2022-05-05T18:42:03+0000
HOST: srv1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.192.64.254              0.0%    10    0.2   0.3   0.2   0.6   0.1
  2.|-- 10.192.80.187              0.0%    10    0.2   0.2   0.2   0.2   0.0

IPv6

root@srv1:~# mtr --address 2620:0:861:11c::129 -n -r 2620:0:861:cabf::187
Start: 2022-05-05T18:41:03+0000
HOST: srv1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:0:861:11c::254        0.0%    10    0.4   0.8   0.3   5.0   1.4
  2.|-- 2620:0:861:cabf::187       0.0%    10    0.2   0.3   0.2   1.1   0.3

Inter-Vlan/subnet routing via IRB interfaces on separate switches

IPv4

From test server 1 in Vlan 2004 on leaf1 ping test server2 in vlan 2005 on leaf2

root@srv1:~# mtr --address 10.192.64.10 -n -r -c 1 10.192.80.10
Start: 2022-04-28T10:03:53+0000
HOST: srv1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.192.64.254              0.0%     1    0.4   0.4   0.4   0.4   0.0
  2.|-- 10.192.80.254              0.0%     1    0.4   0.4   0.4   0.4   0.0
  3.|-- 10.192.80.10               0.0%     1    0.3   0.3   0.3   0.3   0.0

IPv6

Routing works as expected, we can ping between subnets across switches just fine:

root@srv1:~# ping -I 2620:0:861:11c::10 2620:0:861:cabf::10
PING 2620:0:861:cabf::10(2620:0:861:cabf::10) from 2620:0:861:11c::10 : 56 data bytes
64 bytes from 2620:0:861:cabf::10: icmp_seq=1 ttl=62 time=0.227 ms
64 bytes from 2620:0:861:cabf::10: icmp_seq=2 ttl=62 time=0.176 ms

One issue we note, however, is that when doing a traceroute the *remote* switch shows up with an IPv6 link-local address.

root@srv1:~# mtr --address 2620:0:861:11c::10 -n -r 2620:0:861:cabf::10
Start: 2022-04-28T10:14:45+0000
HOST: srv1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:0:861:11c::254        0.0%    10    0.4   0.4   0.3   0.4   0.0
  2.|-- fe80::3e2c:30ff:fe4c:8183  0.0%    10    0.3   0.8   0.3   4.9   1.4
  3.|-- 2620:0:861:cabf::10        0.0%    10    0.3   0.3   0.3   0.3   0.0

The equivalent at hop 2 in the IPv4 test was the address of the Vlan2005 interface on Leaf 2 (10.192.80.254). That interface is configured with a global unicast IPv6 address, and v6 is enabled, but it doesn't use it:

interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 ip anycast-address 10.192.80.254/22
 ipv6 anycast-address 2620:0:861:cabf::254/64

Looking a little closer it seems the address used to source the ICMPv6 PTB messages is the one assigned to Vlan4000 on the switch (this is the Vlan created to bind to our VRF for VXLAN encap):

admin@dell-leaf2:~$ ip -br addr show  | grep 8183
Vlan4000@Bridge  UP             fe80::3e2c:30ff:fe4c:8183/64 
dell-leaf2# show running-configuration interface Vlan 4000
!
interface Vlan4000
 description "IRB VLAN"
 ip vrf forwarding Vrf_codfw
 ipv6 enable


We see the same behaviour when doing a trace which routes out via an external device connected to the Spine layer within the VRF:

root@srv1:~# mtr --address 2620:0:861:11c::10 -n -r 2001:67c:930:400::26 
Start: 2022-04-28T10:39:04+0000
HOST: srv1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:0:861:11c::254        0.0%    10    0.3   0.3   0.3   0.3   0.0
  2.|-- fe80::e29:efff:fee1:fb01   0.0%    10    0.5   0.4   0.4   0.5   0.0
  3.|-- 2001:67c:930:400::26       0.0%    10    5.5   6.1   2.0  11.1   2.4

In this case again the IP used by Spine1 (hop 2) is link local. Looking at the device this IP is seemingly used on both the external interface and the VLan4000 device:

admin@dell-spine1:~$ ip -br addr show | grep "fee1:fb01"
Ethernet104      UP             172.16.1.25/30 2001:67c:930:400::25/64 fe80::e29:efff:fee1:fb01/64 
Vlan4000@Bridge  UP             fe80::e29:efff:fee1:fb01/64 

It certainly would be better if the system had of used global unicast address 2001:67c:930:400::25, or another unicast address on the device, to source the ICMP PTB messages. In contrast the Juniper QFX paltform does use global unicast addressing to source ICMPs, which makes troubleshooting easier as the switches can be identified in a trace:

cmooney@elastic1089:~$ sudo traceroute -I -w 1 -s 2620:0:861:109:10:64:130:7 2620:0:861:10d:10:64:134:2
traceroute to 2620:0:861:10d:10:64:134:2 (2620:0:861:10d:10:64:134:2), 30 hops max, 80 byte packets
 1  irb-1031.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:109::1)  0.610 ms  0.583 ms  0.576 ms
 2  irb-1035.lsw1-f1-eqiad.eqiad.wmnet (2620:0:861:10d::1)  0.515 ms  0.509 ms  0.504 ms
 3  dumpsdata1007.eqiad.wmnet (2620:0:861:10d:10:64:134:2)  0.170 ms * *

IPv6 RA Generation from Vlan Interfaces

NOT WORKING

Vlan configuration as follows on Leaf2:

interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 ip anycast-address 10.192.80.254/22
 ipv6 anycast-address 2620:0:861:cabf::254/6

No IPv6 RAs are recieved on SRV2 which is connected on an access port in this Vlan however:

root@srv2:~# tcpdump -i enp59s0f0 -l -nn icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

BGP Peering on Vlan segment to end device

IPv4 only

Srv1 is reachable in Vlan2004 on IP 10.192.64.10

dell-leaf1# show ip arp vrf Vrf_codfw 
Type: R - Remote Neighbor entries (EVPN)
-------------------------------------------------------------------------------------------
Address             Hardware address    Interface            Egress Interface    Type
-------------------------------------------------------------------------------------------
10.192.64.10        40:a8:f0:2c:83:10   Vlan2004             Ethernet0           Dynamic  

We configure a BGP session to it within the VRF as follows:

router bgp 65032 vrf Vrf_codfw
 router-id 10.0.1.24
 log-neighbor-changes
 bestpath as-path multipath-relax
 !
 address-family ipv4 unicast
  redistribute connected 
  maximum-paths 16
 !
 neighbor 10.192.64.10
  remote-as 64600
  timers 5 20
  advertisement-interval 0
  bfd
  local-as 14907 no-prepend replace-as
  !
  address-family ipv4 unicast
   activate
   route-map NO_HOST_ROUTES out
   remove-private-as
  !

The "NO_HOST_ROUTES" filter is designed to prevent /32 host routes from being sent externally. These end up in the VRF BGP RIB due to the import of EVPN type 2 MAC/IP routes to the local table. Instead we only want to send the routes that have originated locally or as type 5 EVPN routes, i.e. the subnets allocated to our Vlans and not every host IP from the Vlan.

ip prefix-list NO_HOST_ROUTES seq 5 permit 0.0.0.0/0 le 29
route-map NO_HOST_ROUTES permit 100
 match ip address prefix-list NO_HOST_ROUTES

Srv1 is set up for BGP (running FRR in this case), configured to peer with the IP of Leaf1 on Vlan2004:

router bgp 64600
 neighbor 10.192.64.254 remote-as 14907
 neighbor 10.192.64.254 bfd
 !
 address-family ipv4 unicast
 exit-address-family

With this in place the session establishes:

srv1# show bgp ipv4 unicast summary 
BGP router identifier 1.1.1.1, local AS number 64600 vrf-id 0
BGP table version 15
RIB entries 5, using 920 bytes of memory
Peers 1, using 723 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.192.64.254   4      14907       482       396        0    0    0 00:21:21            2        1 N/A

Total number of neighbors 1

The prefix for Vlan2004 is originated locally by leaf1 (due to the 'redistributed connected'), the prefix for Vlan2005 is in the BGP RIB on Leaf1 already having been redistributed from the EVPN. Both are learnt on the server side:

srv1# show bgp ipv4 unicast neighbors 10.192.64.254 routes 
BGP table version is 24, local router ID is 1.1.1.1, vrf id 0
Default local pref 100, local AS 64600
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.192.64.0/22   10.192.64.254            0             0 14907 ?
*> 10.192.80.0/22   10.192.64.254                          0 14907 ?

The as-path just shows the "fake" AS configured in the 'local-as' command, as desired. The Spine and Leaf2 AS numbers would appear for 10.192.80.0/22 but we have configured the sessions to "remove-private-as" to validate that works as expected.

IPv4 carrying IPv4 & IPv6 address families

Configuration on the switch was the same as the last test, but the IPv6 unicast address family was also activated for the server peer:

router bgp 65032 vrf Vrf_codfw
 neighbor 10.192.64.10
  remote-as 64600
  !
  address-family ipv6 unicast
   activate
   remove-private-as
!

With the same done server-side the BGP session restarted, and came back up announcing IPv6 addresses also:

dell-leaf1# show bgp ipv6 unicast vrf Vrf_codfw summary
BGP router identifier 10.0.1.24, local AS number 65032 
Neighbor         V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd   
10.192.64.10     4   64600   144556    173488    0       0       00:06:06        1              
 
Total number of neighbors 1
Total number of neighbors established 1

Address was received as expected:

dell-leaf1# show bgp ipv6 unicast vrf Vrf_codfw neighbors 10.192.64.10 routes 
BGP routing table information for VRF default
Router identifier 10.0.1.24, local AS number 65032 
Route status codes: * - valid, > - best
Origin codes: i - IGP, e - EGP, ? - incomplete
     Network             Next Hop            Metric      LocPref     Weight Path
*>   2620:0:861:babe::1/1282620:0:861:11c::10  0                              64600 ?
Total number of prefixes 1

And placed into the local VRF routing table as required:

dell-leaf1# show ipv6 route vrf Vrf_codfw 2620:0:861:babe::1
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF
        > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
       Destination                  Gateway                                                 Dist/Metric   Uptime      
--------------------------------------------------------------------------------------------------------------------
 B>*   2620:0:861:babe::1/128       via fe80::42a8:f0ff:fe2c:8310 Vlan2004                  20/0          00:05:50

IPv4 & IPv6 each carrying their own address family

BGP Peering on Vlan segment from Anycast GW IP

IPv4 only

IPv4 carrying IPv4 & IPv6 address families

IPv4 & IPv6 each carrying their own address family

BGP Peering on Vlan segment with Anycast GW, from Unicast IP

NOT WORKING

It does not seem possible to configure an additional IP address on a Vlan interface configured with an Anycast GW.

For instance with this config on Vlan2005 on Leaf2:

interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.80.254/22
 ipv6 anycast-address 2620:0:861:cabf::254/64

We cannot add an additional (unique to this switch) IP address in addition to the anycast IP (used on all switches):

dell-leaf2# configure terminal
dell-leaf2(config)# interface Vlan2005
dell-leaf2(conf-if-Vlan2005)# ip address 10.192.80.253/22 secondary
%Error: Primary IPv4 address is not configured for interface: Vlan2005
dell-leaf2(conf-if-Vlan2005)# ip address 10.192.80.253/22
%Error: IP overlap on same interface with IP or IP Anycast 10.192.80.254/22

This is not something that makes much of a difference to us generally. We can likely create unique loopback interfaces, in the overlay VRF, on the switch with unique IPs per device, if needed. The only scenario this is probably needed is if we have an MC-LAG configured, and need to peer to both switches over the bonded link. This is not something that is currently likely for us to require.

BGP Re-establishment if device peered to Anycast GW moved to new switch (i.e. VM live motion)

eBGP Peering in VRF to external device

The VRF was enabled on the spine switches, each of which was connected to a Juniper device to test eBGP routing to an external element from the VRF.

Interface config on Spine1, connected to the Juniper switch was as follows:

dell-spine1# show running-configuration interface Eth1/27
!
interface Eth1/27
 description link_to_lsw3-et-0/0/50
 mtu 9100
 speed 40000
 fec none
 no shutdown
 ip vrf forwarding Vrf_codfw
 ip address 172.16.1.25/30
 ipv6 address 2001:67c:930:400::25/64
 ipv6 enable

BGP sessions were configured, one over IPv4 for IPv4 address-fam, one over IPv6 for IPv6 address-fam:

router bgp 65030 vrf Vrf_codfw
 router-id 10.0.1.13
 !
 address-family ipv4 unicast
  maximum-paths 8
  maximum-paths ibgp 8
 !
 address-family ipv6 unicast
  maximum-paths 8
  maximum-paths ibgp 8
 !
 neighbor 172.16.1.26
  remote-as 65034
  !
  address-family ipv4 unicast
   activate
 !
 neighbor 2001:67c:930:400::26
  remote-as 65034
  !
  address-family ipv4 unicast
  !
  address-family ipv6 unicast
   activate

BGP established as expected for each address family:

dell-spine1# show bgp ipv4 unicast vrf Vrf_codfw summary
BGP router identifier 10.0.1.13, local AS number 65030
Neighbor        V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd
172.16.1.26     4   65034   24829     22586     0       0       01w0d20h        1  
dell-spine1# show bgp ipv6 unicast vrf Vrf_codfw summary
BGP router identifier 10.0.1.13, local AS number 65030
Neighbor                 V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd
2001:67c:930:400::26     4   65034   167       150       0       0       01:04:40        1

Routes are accepted:

dell-spine1# show bgp ipv4 unicast vrf Vrf_codfw neighbors 172.16.1.26 routes
BGP routing table information for VRF default
Router identifier 10.0.1.13, local AS number 65030
Route status codes: * - valid, > - best
Origin codes: i - IGP, e - EGP, ? - incomplete
     Network             Next Hop            Metric      LocPref     Weight Path
*>   0.0.0.0/0           172.16.1.26                                        65034 i
Total number of prefixes 1
dell-spine1# show bgp ipv6 unicast vrf Vrf_codfw neighbors 2001:67c:930:400::26 routes
BGP routing table information for VRF default
Router identifier 10.0.1.13, local AS number 65030
Route status codes: * - valid, > - best
Origin codes: i - IGP, e - EGP, ? - incomplete
     Network             Next Hop            Metric      LocPref     Weight Path
*>   ::/0                2001:67c:930:400::26                               65034 i
Total number of prefixes 1

And make their way into the local FIB:

dell-spine1# show ip route vrf Vrf_codfw 0.0.0.0/0
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF
        > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
       Destination                  Gateway                                                 Dist/Metric   Uptime
--------------------------------------------------------------------------------------------------------------------
 B>*   0.0.0.0/0                    via 172.16.1.26               Eth1/27                   20/0          01w0d20h


dell-spine1# show ipv6 route vrf Vrf_codfw ::/0
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF
        > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
       Destination                  Gateway                                                 Dist/Metric   Uptime
--------------------------------------------------------------------------------------------------------------------
 B>*   ::/0                         via fe80::2e21:31ff:fefa:f1bb Eth1/27                   20/0          01:04:58

BFD Support on Peering to External Device (VRF)

BFD was enabled for the peering to srv1 in Vlan2004 as follows on the switch:

router bgp 65032 vrf Vrf_codfw
 neighbor 10.192.64.10
  remote-as 64600
  bfd

Once it was configured on the server as well the session came up:

dell-leaf1# show bfd peers vrf Vrf_codfw
BFD Peers:
     
    peer 10.192.64.10 vrf Vrf_codfw interface Vlan2004
        ID: 624049837
        Remote ID: 3910201229
        Status: up
        Uptime: 0 day(s), 0 hour(s), 0 min(s), 46 sec(s)
        Diagnostics: ok
        Remote diagnostics: ok
        Peer Type: dynamic
        Local timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo transmission interval: 0ms
        Remote timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo transmission interval: 50ms

BFD Support on BGP peering in global table

BFD was set up in the underlay network / global table on the BGP peerings between Spine1 and the two Leaf devices. Simply needs the keyword 'bfd' added under the BGP neighbor definition:

router bgp 65030
 neighbor 172.16.1.10
  remote-as 65033
  bfd

Once configured both sides the sessions come up:

dell-spine1# show bfd peers brief 
Session Count: 2
SessionId  LocalAddress                              PeerAddress                               Status         Vrf     
=========  ============                              ===========                               ======         ===     
2747556315 172.16.1.9                                172.16.1.10                               UP             default 
1863681777 172.16.1.1                                172.16.1.2                                UP             default 
dell-spine1# show bfd peers 
BFD Peers:
     
   
    peer 172.16.1.10 vrf default interface Eth1/31
        ID: 2747556315
        Remote ID: 1404963392
        Status: up
        Uptime: 0 day(s), 0 hour(s), 1 min(s), 27 sec(s)
        Diagnostics: ok
        Remote diagnostics: ok
        Peer Type: dynamic
        Local timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo transmission interval: 0ms
        Remote timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo transmission interval: 50ms
     
   
    peer 172.16.1.2 vrf default interface Eth1/32
        ID: 1863681777
        Remote ID: 2448635902
        Status: up
        Uptime: 0 day(s), 0 hour(s), 5 min(s), 53 sec(s)
        Diagnostics: ok
        Remote diagnostics: ok
        Peer Type: dynamic
        Local timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo transmission interval: 0ms
        Remote timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo transmission interval: 50ms

BGP Route propagaton from unicast peer into EVPN and into remote VRF table

IPv4

We learn 1.1.1.1/32 from srv1 on Leaf1:

dell-leaf1# show bgp ipv4 unicast vrf Vrf_codfw neighbors 10.192.64.10 routes 
BGP routing table information for VRF default
Router identifier 10.0.1.24, local AS number 65032 
Route status codes: * - valid, > - best
Origin codes: i - IGP, e - EGP, ? - incomplete
     Network             Next Hop            Metric      LocPref     Weight Path
*>   1.1.1.1/32          10.192.64.10        0                              64600 ?
Total number of prefixes 1

We can see this being learnt on Leaf 2 as an EVPN type-5 route, next-hop the Leaf1 VTEP IP:

dell-leaf2# show bgp l2vpn evpn route detail | find 1.1.1.1
BGP routing table entry for 10.0.1.24:5096:[5]:[0]:[32]:[1.1.1.1]
Paths: (2 available, best #1)
  Advertised to non peer-group peers:
  172.16.1.9 172.16.1.13
  Route [5]:[0]:[32]:[1.1.1.1] VNI 404000
  65030 65032 64600
    10.10.10.1 from 172.16.1.9 (10.0.1.13)
      Origin incomplete, valid, external, best (Router ID)
      Extended Community: RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
      Last update: Fri Apr 15 15:14:11 2022
  Route [5]:[0]:[32]:[1.1.1.1] VNI 404000
  65030 65032 64600
    10.10.10.1 from 172.16.1.13 (10.0.1.14)
      Origin incomplete, valid, external
      Extended Community: RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
      Last update: Fri Apr 15 15:14:11 2022

This is correctly added to the local VRF routing table on Leaf 2:

dell-leaf2# show ip route vrf Vrf_codfw 1.1.1.1
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF
        > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
       Destination                  Gateway                                                 Dist/Metric   Uptime      
--------------------------------------------------------------------------------------------------------------------
 B>*   1.1.1.1/32                   via 10.10.10.1                Vlan4000                  20/0          00:26:49   

We can send packets from srv2 (connected to Leaf 2 on Vlan2005) and get a response from 1.1.1.1 on srv1 (connected to Leaf1):

ppaul@srv2:~$ mtr -n -r 1.1.1.1 -c 3
Start: 2022-04-15T15:30:42+0000
HOST: srv2                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.192.80.254              0.0%     3    0.2   0.3   0.2   0.3   0.0
  2.|-- 10.192.64.254              0.0%     3    0.3   0.3   0.3   0.3   0.0
  3.|-- 1.1.1.1                    0.0%     3    0.3   0.3   0.3   0.3   0.0

IPv6

We learn 2620:0:861:babe::1/128 via BGP from srv1 connected over Vlan2004 on Leaf1:

dell-leaf1# show bgp ipv6 unicast vrf Vrf_codfw neighbors 10.192.64.10 routes
BGP routing table information for VRF default
Router identifier 10.0.1.24, local AS number 65032 
Route status codes: * - valid, > - best
Origin codes: i - IGP, e - EGP, ? - incomplete
     Network             Next Hop            Metric      LocPref     Weight Path
*>   2620:0:861:babe::1/1282620:0:861:11c::10  0                              64600 ?
Total number of prefixes 1

This is received on Leaf 2 as an EVPN type-5 with next-hop of Leaf1's VTEP IP:

dell-leaf2# show bgp l2vpn evpn route detail | find babe
BGP routing table entry for 10.0.1.24:5096:[5]:[0]:[128]:[2620:0:861:babe::1]
Paths: (2 available, best #1)
  Advertised to non peer-group peers:
  172.16.1.9 172.16.1.13
  Route [5]:[0]:[128]:[2620:0:861:babe::1] VNI 404000
  65030 65032 64600
    10.10.10.1 from 172.16.1.9 (10.0.1.13)
      Origin incomplete, valid, external, best (Router ID)
      Extended Community: RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
      Last update: Fri Apr 15 15:28:43 2022
  Route [5]:[0]:[128]:[2620:0:861:babe::1] VNI 404000
  65030 65032 64600
    10.10.10.1 from 172.16.1.13 (10.0.1.14)
      Origin incomplete, valid, external
      Extended Community: RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
      Last update: Fri Apr 15 15:28:43 2022

It's added to the local VRF routing table on Leaf 2:

dell-leaf2# show ipv6 route vrf Vrf_codfw 2620:0:861:babe::1
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF
        > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
       Destination                  Gateway                                                 Dist/Metric   Uptime      
--------------------------------------------------------------------------------------------------------------------
 B>*   2620:0:861:babe::1/128       via ::ffff:10.10.10.1         Vlan4000                  20/0          00:11:10  


And we can send traffic to it from srv2 (connected to Leaf2), similar to the v4 test:

root@srv2:~# mtr -n -r -c 3 2620:0:861:babe::1
Start: 2022-04-15T15:37:51+0000
HOST: srv2                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:0:861:cabf::254       0.0%     3    0.4   0.4   0.4   0.4   0.0
  2.|-- fe80::3e2c:30ff:fe4b:903   0.0%     3    0.3   0.3   0.3   0.4   0.0
  3.|-- 2620:0:861:babe::1         0.0%     3    0.3   0.3   0.3   0.3   0.0

BGP Route propagation to external hosts from VRF

IPv4

BGP peering in vrf Vrf_codfw to srv1 was set up the same way as in test 3.15.

On the server we can see that we learn the routes for Vlan2004 (originated by leaf1) and Vlan2005 (originated by leaf2 and propagated as an EVPN type 5 to leaf 1):

srv1# show bgp ipv4 unicast neighbors 10.192.64.254 routes 
BGP table version is 15, local router ID is 1.1.1.1, vrf id 0
Default local pref 100, local AS 64600
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.192.64.0/22   10.192.64.254            0             0 14907 65032 ?
*> 10.192.80.0/22   10.192.64.254                          0 14907 65032 65030 65033 ?


IPv6

With IPv6 peering to srv1 we see the same thing, we learn the /64 subnets attached to Vlan2004 (local on BGP peer Leaf 1) and also Vlan 2005 (on Leaf 2, and learnt by Leaf 1 over EVPN):

srv1# show bgp ipv6 unicast neighbors 10.192.64.254 routes 
BGP table version is 10, local router ID is 1.1.1.1, vrf id 0
Default local pref 100, local AS 64600
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 2620:0:861:11c::/64
                    fe80::200:ff:fe10:1010
                                             0             0 14907 ?
*> 2620:0:861:cabf::/64
                    fe80::200:ff:fe10:1010
                                                           0 14907 65030 65033 ?

ECMP Routing in overlay network

In modern Leaf/Spine fabrics ECMP is a key requirement to make use of multiple redundant links. To test this is working correctly for routes learnt in EVPN type 5 announcements we augmented the configuration described in section 3.19 "eBGP Peering in VRF to external device".

In that section an external device was connected to Spine1 and BGP peering established within the VRF. For this test we connected that same device to Spine2 also, and configured BGP peering from Spine2 for it, announcing the same default routes. Interface and BGP configuration was similar, and BGP established as in that test:

dell-spine2# show bgp ipv4 unicast vrf Vrf_codfw summary
BGP router identifier 10.0.1.14, local AS number 65030 
Neighbor        V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd   
172.16.1.30     4   65034   24762     22530     0       0       00:00:46        1  

dell-spine2# show bgp ipv4 unicast vrf Vrf_codfw neighbors 172.16.1.30 routes 
BGP routing table information for VRF default
Router identifier 10.0.1.14, local AS number 65030 
Route status codes: * - valid, > - best
Origin codes: i - IGP, e - EGP, ? - incomplete
     Network             Next Hop            Metric      LocPref     Weight Path
*>   0.0.0.0/0           172.16.1.30                                        65034 i


dell-spine2# show bgp ipv6 unicast vrf Vrf_codfw summary
BGP router identifier 10.0.1.14, local AS number 65030 
Neighbor                 V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd   
2001:67c:930:401::30     4   65034   24        37        0       0       00:00:23        1              
 
Total number of neighbors 1
Total number of neighbors established 1

dell-spine2# show bgp ipv6 unicast vrf Vrf_codfw neighbors 2001:67c:930:401::30 routes 
BGP routing table information for VRF default
Router identifier 10.0.1.14, local AS number 65030 
Route status codes: * - valid, > - best
Origin codes: i - IGP, e - EGP, ? - incomplete
     Network             Next Hop            Metric      LocPref     Weight Path
*>   ::/0                2001:67c:930:401::30                               65034 i


With that set up we now have a situation where Spine1 and Spine2 are learning a default route in each address family, from an external peer. These should then get announced to the Leaf devices as separate EVPN type 5 prefixes, with next-hop set to the VTEP IP of Spine1 and Spine2. Ultimately the LEAF devices should then ECMP traffic to both spines based on these two routes.

Looking on Leaf2 we can see this is the case, we have 2 v4 and 2 v6 addresses, 1 of each from each spine:

dell-leaf2# show bgp l2vpn evpn route type prefix
BGP table version is 2, local router ID is 10.0.1.25
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
   Network          Next Hop            Metric LocPrf Weight Path
                    Extended Community
Route Distinguisher: 10.0.1.13:5096
*>   [5]:[0]:[0]:[0.0.0.0]
                    10.10.10.3                                     0 65030 65034 i
                    RT:65030:404000 ET:8 Rmac:0c:29:ef:e1:fb:01
*>   [5]:[0]:[0]:[::] 10.10.10.3                                     0 65030 65034 i
                    RT:65030:404000 ET:8 Rmac:0c:29:ef:e1:fb:01
Route Distinguisher: 10.0.1.14:5096
*>   [5]:[0]:[0]:[0.0.0.0]
                    10.10.10.4                                     0 65030 65034 i
                    RT:65030:404000 ET:8 Rmac:0c:29:ef:e1:fc:81
*>   [5]:[0]:[0]:[::] 10.10.10.4                                     0 65030 65034 i
                    RT:65030:404000 ET:8 Rmac:0c:29:ef:e1:fc:81


IPv4

Sticking with Leaf2 if we look at the local VRF routing table it does indeed have both defaults, and both are in the FIB indicating it should ECMP across the two next-hops:

dell-leaf2# show ip route vrf Vrf_codfw 0.0.0.0/0
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF
        > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
       Destination                  Gateway                                                 Dist/Metric   Uptime
--------------------------------------------------------------------------------------------------------------------
 B>*   0.0.0.0/0                    via 10.10.10.3                Vlan4000                  20/0          00:09:44
   *                                via 10.10.10.4                Vlan4000

If we move to SRV2, which is connected on an access port to Leaf2, we can see the different link addresses of the connections from Spine1 or Spine2 in a traceroute if we change the source address (and thus the ECMP hash):

root@srv2:~# mtr --address 10.192.80.10 -n -r -c 3 10.0.1.26
Start: 2022-04-28T12:15:45+0000
HOST: srv2                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.192.80.254              0.0%     3    0.3   0.3   0.3   0.3   0.0
  2.|-- 172.16.1.29                0.0%     3    0.3   0.4   0.3   0.4   0.0
  3.|-- 10.0.1.26                  0.0%     3   21.2  18.8  13.7  21.5   4.4

^ Hop 2 is Spine2 in this case.

root@srv2:~# mtr --address 10.192.80.21 -n -r -c 3 10.0.1.26
Start: 2022-04-28T12:16:01+0000
HOST: srv2                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.192.80.254              0.0%     3    0.3   0.3   0.3   0.3   0.0
  2.|-- 172.16.1.25                0.0%     3    0.3   0.3   0.3   0.4   0.0
  3.|-- 10.0.1.26                  0.0%     3   71.6  32.4  11.4  71.6  33.9


^ Hop 2 is Spine1 in this case.

IPv6

Again if we look on Leaf2 the two EVPN routes have been added to the local VRF table and it indicates both are being used:

dell-leaf2# show ipv6 route vrf Vrf_codfw ::/0
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF
        > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
       Destination                  Gateway                                                 Dist/Metric   Uptime      
--------------------------------------------------------------------------------------------------------------------
 B>*   ::/0                         via ::ffff:10.10.10.3         Vlan4000                  20/0          00:15:08    
   *                                via ::ffff:10.10.10.4         Vlan4000                             

Doing multiple traceroutes, changing the source IP address, we can see different IPs at hop 2, depending on if the packets are sent by Leaf2 to Spine1 or Spine2:

Source: 2620:0:861:cabf::66
Start: 2022-04-28T12:34:11+0000
HOST: srv2                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:0:861:cabf::254       0.0%     2    0.3   0.5   0.3   0.6   0.2
  2.|-- fe80::e29:efff:fee1:fb01   0.0%     2    0.4   0.4   0.4   0.5   0.1
  3.|-- 2606:4700:4700::1111       0.0%     2    4.2   5.9   4.2   7.7   2.4
Source: 2620:0:861:cabf::67
Start: 2022-04-28T12:34:17+0000
HOST: srv2                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:0:861:cabf::254       0.0%     2    0.3   0.5   0.3   0.6   0.2
  2.|-- fe80::e29:efff:fee1:fb01   0.0%     2    0.4   0.4   0.4   0.4   0.1
  3.|-- 2606:4700:4700::1111       0.0%     2   25.1  14.9   4.7  25.1  14.4
Source: 2620:0:861:cabf::68
Start: 2022-04-28T12:34:24+0000
HOST: srv2                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:0:861:cabf::254       0.0%     2    0.3   0.4   0.3   0.6   0.2
  2.|-- fe80::e29:efff:fee1:fc81   0.0%     2    0.4   0.4   0.4   0.4   0.0
  3.|-- 2606:4700:4700::1111       0.0%     2    4.3   4.6   4.3   4.9   0.4
Source: 2620:0:861:cabf::69
Start: 2022-04-28T12:34:30+0000
HOST: srv2                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:0:861:cabf::254       0.0%     2    0.4   0.5   0.4   0.7   0.2
  2.|-- fe80::e29:efff:fee1:fc81   0.0%     2    0.4   0.4   0.4   0.5   0.1
  3.|-- 2606:4700:4700::1111       0.0%     2    9.4   8.7   8.0   9.4   1.0

ARP Supression

ND Suppression

DHCP Relay & Option 82 insertion

DHCP Relay

DHCP relay can be configured on a Vlan interface, for instance like this on Vlan 2005 on Leaf2:

interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ip dhcp-relay 10.192.64.10 vrf Vrf_codfw
 ip anycast-address 10.192.80.254/22

In the above case the 10.192.64.10 IP is configured on SRV1, which is connected to Vlan 2004 on Leaf1.

On SRV2, which is connected via an access port to Leaf2 on Vlan2005, we can then issue a DHCP request, which completes as desired:

root@srv2:~# dhclient -v enp59s0f0
Internet Systems Consortium DHCP Client 4.3.5
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/enp59s0f0/40:a8:f0:2c:31:68
Sending on   LPF/enp59s0f0/40:a8:f0:2c:31:68
Sending on   Socket/fallback
DHCPDISCOVER on enp59s0f0 to 255.255.255.255 port 67 interval 3 (xid=0xe029ca10)
DHCPREQUEST of 10.192.80.129 on enp59s0f0 to 255.255.255.255 port 67 (xid=0x10ca29e0)
DHCPOFFER of 10.192.80.129 from 10.192.80.254
DHCPACK of 10.192.80.129 from 10.192.80.254
bound to 10.192.80.129 -- renewal in 147 seconds.

Looking on SRV1 we can see that this has been relayed from the IPv4 address on Vlan 2005 of Leaf2:

root@srv1:/etc/dhcp# tcpdump -i enp59s0f0.2004 -l -nn udp port 67 or udp port 68
listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
14:45:58.969588 IP 10.192.80.254.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 300
14:45:59.971058 IP 10.192.64.10.67 > 10.192.80.254.67: BOOTP/DHCP, Reply, length 323
14:46:01.890686 IP 10.192.80.254.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 300
14:46:01.890952 IP 10.192.64.10.67 > 10.192.80.254.67: BOOTP/DHCP, Reply, length 323
14:46:08.511262 IP 10.192.80.254.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 300
14:46:08.511501 IP 10.192.64.10.67 > 10.192.80.254.67: BOOTP/DHCP, Reply, length 323

This worked fine in this case, when Vlan2005 was only configured on Leaf2. We found, however, that when Vlan 2005 was enabled on both switches, and both had the same IP address configured as Anycast GW, it failed. The reason for this is the source IP Leaf2 used to sent the DHCP packets was also configured on Leaf1. So when SRV1 replied to the request Leaf1 tried to process the packet itself, instead of sending it to Leaf2.

Obviously the relayed DHCP packet needs to come from an IP address only configured on the device that sends it, so that the replies go back to the correct device. An Anycast GW IP is thus not suitable, as it is shared on them all. Unfortunately SONiC does not allow a 'secondary', unique, IP to be added in addition to the Anycast GW IP on a Vlan interface. To get around this limitation we created a new loopback interface on Leaf2, assigned it an IP address, and placed it in the VRF:

dell-leaf2# show running-configuration interface Loopback 2
!
interface Loopback 2
 ip vrf forwarding Vrf_codfw
 ip address 1.2.3.4/32

We then added this config to the Vlan interface to tell it to source the DHCP relays from that IP:

interface Vlan2005 
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.80.254/22
 ip dhcp-relay 10.192.64.10 vrf Vrf_codfw
 ip dhcp-relay source-interface Loopback2

With this in place we re-tried the DHCP request from SRV2. We did see the packet come in from source IP 1.2.3.4, but unfortunately the reply went to 10.192.80.254 still:

root@srv1:/etc/dhcp# tcpdump -i enp59s0f0.2004 -l -nn udp port 67 or udp port 68
listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
15:15:05.556947 IP 1.2.3.4.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 302
15:15:06.558359 IP 10.192.64.10.67 > 10.192.80.254.67: BOOTP/DHCP, Reply, length 323

Further investigation revealed that we could also add this command to the Vlan interface configuration:

interface Vlan2005 
 ip dhcp-relay 10.192.64.10 vrf Vrf_codfw
 ip dhcp-relay source-interface Loopback2
 ip dhcp-relay link-select

This command caused two additional DHCP Option 82 elements to be added to the packet:

Option 82 Suboption: (5) Link selection
    Length: 4
    Link selection: 10.192.80.254
Option 82 Suboption: (11) Server ID Override
    Length: 4
    Server ID Override: 10.192.80.254

Our vanilla ISC DHCPd test server replied to the source IP of the packet correctly when these attributes were present, and the replies went back to Leaf2 correctly and ultimately the end server:

root@srv1:~# tcpdump -i enp59s0f0.2004 -l -nn udp port 67 or udp port 68
listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
15:12:02.011778 IP 1.2.3.4.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 314
15:12:02.011987 IP 10.192.64.10.67 > 1.2.3.4.67: BOOTP/DHCP, Reply, length 335

So if we use Anycast GW across multiple switches we need to create a switch-specific Loopback interface, each with a unique IP address, and add this as the "ip dhcp-relay source-interface" as well as enable "ip dhcp-relay link-select".

Option 82

The SONiC devices do insert Option 82 into DHCP messages they relay, as can be seen in the following snippet from a packet capture:

Option: (82) Agent Information Option
   Length: 29
   Option 82 Suboption: (1) Agent Circuit ID
       Length: 8
       Agent Circuit ID: 566c616e32303035
   Option 82 Suboption: (2) Agent Remote ID
       Length: 17
       Agent Remote ID: 30303a30303a30303a31303a31303a3130

Decoding these values we see the following information:

Agent Circuit ID: Vlan2005
Agent Remote ID:  00:00:00:10:10:10

The remote ID is the MAC address of the Vlan2005 interface on Leaf2 which sourced the packet:

admin@dell-leaf2:~$ ip -br link show Vlan2005
Vlan2005@Bridge  UP             00:00:00:10:10:10 <BROADCAST,MULTICAST,UP,LOWER_UP> 

Unfortunately the SONiC config does not seem to provide any mechanism to customize what is included in these values. Specifically it does not seem to allow us to include the switch name, and access port ID the DHCP request was received on, which our install process currently uses to identify the server and assign the correct IP.

IP Filters on Routed interface

IPv4

IPv6

IP Filters on IRB interface

IPv4

IPv6

Filter access to RE/CPU/Device Services

Evaluate if there is a mechanism like the loopback filter in JunOS, or specific daemon filters (SNMP, vty ACL, NTP acls etc) on Cisco. Basically to prevent remote users trying to SSH to switch or similar. If nothing specific it can be done in-band on the external data ports, but it's trickier to implement.

Failover Tests

Spine Switch Failure

Leaf to Spine Link Failure

Management Tests

User Account Creation

SSH Access to Management

SSH Key Auth

Management VRF

We should attempt to place the dedicated management port in a specific VRF ("mgmt" for instance).

Ideally all following tests would be configured with this, and all functions would work / have a way to configure them to work when access is in mgmt vrf.

Interface    IPv4 address/mask    Master     Admin/Oper    BGP Neighbor    Neighbor IP    Flags
-----------  -------------------  ---------  ------------  --------------  -------------  -------
Ethernet72   172.16.1.6/30                   up/up         N/A             N/A
Ethernet76   172.16.1.2/30                   up/up         N/A             N/A
Loopback0    10.0.1.24/32                    up/up         N/A             N/A
Loopback1    10.10.10.1/32                   up/up         N/A             N/A
Vlan2004     10.192.64.254/22     Vrf_codfw  up/up         N/A             N/A            A
Vlan2005     10.192.80.254/22     Vrf_codfw  down/down     N/A             N/A            A
docker0      240.127.1.1/24                  up/down       N/A             N/A
eth0         10.193.0.185/16      mgmt       up/up         N/A             N/A

SNMP RO Access

Works fine, devices added to LibreNMS system which is polling them via SNMP fine.

File:Dellasw snmp.png

NTP

Switch was configured to act as an NTP client in the mangement VRF towards our NTP servers:

   dell-leaf2# show running-configuration | grep ntp
   ntp server 208.80.153.77 minpoll 6 maxpoll 10
   ntp server 208.80.154.10 minpoll 6 maxpoll 10
   ntp server 208.80.155.108 minpoll 6 maxpoll 10
   ntp vrf mgmt

Following this NTP sync was ok:

   dell-leaf2# show ntp associations 
   remote                      refid            st   t  when   poll   reach  delay  offset       jitter      
   ------------------------------------------------------------------------------------------------------ 
    208.80.153.77              162.159.200.1    4    u  27     64     3      0.299  4.204        0.426        
   *208.80.154.10              170.187.158.81   3    u  40     64     17     31.728 0.097        2.565        
    208.80.155.108             104.171.113.34   3    u  31     64     3      33.109 4.581        0.419       
   ------------------------------------------------------------------------------------------------------
   * master (synced), # master (unsynced), + selected, - candidate, ~ configured

LLDP

Server Side

Server can see the switch and gets the switch port ok:

   root@srv1:~# lldpcli show neighbors ports eno1
   -------------------------------------------------------------------------------
   LLDP neighbors:
   -------------------------------------------------------------------------------
   Interface:    eno1, via: LLDP, RID: 1, Time: 0 day, 00:00:01
     Chassis:     
       ChassisID:    mac 3c:2c:30:4b:09:00
       SysName:      dell-leaf1
       SysDescr:     SONiC Software Version: SONiC.3.4.1-Enterprise_Base - HwSku: DellEMC-S5248f-P-25G-DPB - Distribution: Debian 9.13 - Kernel: 4.9.0-11-2-amd64
       MgmtIP:       10.193.0.185
       Capability:   Bridge, off
       Capability:   Router, on
       Capability:   Wlan, off
       Capability:   Station, on
     Port:        
       PortID:       local Ethernet32
       PortDescr:    test_srv1_1GTBase
       TTL:          120
   -------------------------------------------------------------------------------

Switch Side

The switch can also see the server details:

   dell-leaf1# show lldp neighbor Ethernet 32
   -----------------------------------------------------------
   LLDP Neighbors      
   -----------------------------------------------------------
   Interface:   Ethernet32,via: LLDP
     Chassis:
       ChassisID:    d0:94:66:86:4d:6c
       SysName:      srv1
       SysDescr:     Ubuntu 18.04.6 LTS Linux 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64
       TTL:          120
       MgmtIP:       1.1.1.1
       MgmtIP:       2620:0:861:11c::10
     Port
       PortID:       d0:94:66:86:4d:6c
       PortDescr:    eno1
   -----------------------------------------------------------

Per-interface LLDP control

The command "no lldp enable" was added to the switch config for Ethernet32, after a short time the switch details were no longer visible from the server:

   root@srv1:~# lldpcli show neighbors ports eno1
   -------------------------------------------------------------------------------
   LLDP neighbors:
   -------------------------------------------------------------------------------
   root@srv1:~# 

Same goes for the switch-side:

   dell-leaf1# show lldp neighbor Ethernet 32
   -----------------------------------------------------------
   LLDP Neighbors      
   -----------------------------------------------------------
   dell-leaf1#

sFlow Export

Set up sFlow export. At a minimum I guess we could just capture the packets with tcpdump and compare to what the Junipers send. We could also try to set up pmacct somewhere or something like that to validate the flow data is ok.


Prometheus Export

Set up telegraf etc. as per their guide. We can test with curl don't need to actually set up our Prometheus to scrape it.


Puppet Agent

It would be interesting to test the puppet agent compatibility. We may not go down that road but good to know.

Automation Tests

RESTCONF

Basic tests to make sure we can talk to the interface and apply config.

Partial config replace

i.e. validate we can do a replace on a specific section, i.e. "bgp", without touching entire config.