You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

SONiC: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Cathal Mooney
imported>Cathal Mooney
No edit summary
 
(4 intermediate revisions by one other user not shown)
Line 22: Line 22:


https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md#configuring-management-interface
https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md#configuring-management-interface
'''Note:''' for the Enterprise Sonic the only commands needed are
<code>ip address 10.193.0.x/16 gwaddr 10.193.0.1 </code>
<code> ip vrf mgmt </code>


'''SNMP'''
'''SNMP'''
Line 38: Line 44:


https://github.com/Azure/SONiC/wiki/How-to-Check-SNMP-Configuration
https://github.com/Azure/SONiC/wiki/How-to-Check-SNMP-Configuration
'''Note:''' for the Enterprise Sonic the only command needed is <code>snmp-server community <YourCommunity></code>
admin@dell-leaf2:~$ sonic-cli
dell-leaf2# configure terminal
dell-leaf2(config)# snmp-server community <yourcommunity>


'''NTP'''
'''NTP'''
Line 46: Line 57:


<code> show ip interface  </code>
<code> show ip interface  </code>
  Interface   Master    IPv4 address/mask   Admin/Oper   BGP Neighbor    Neighbor IP
-----------------------------------------------------------------------------------------------
  ----------- -------- ------------------- ------------ -------------- -------------
  Interface           IP address/mask                   VRF            Admin/Oper     Flags
  Ethernet4              10.0.0.2/31          up/up         ARISTA02T2      10.0.0.3
  -----------------------------------------------------------------------------------------------
  Ethernet8              10.0.0.4/31          up/up         ARISTA03T2      10.0.0.5
  Ethernet120          172.16.1.9/30                                    up/up
  Ethernet12            10.0.0.6/31          up/up         ARISTA04T2      10.0.0.7
  Ethernet124          172.16.1.1/30                                    up/up
  Ethernet16            10.0.0.8/31          up/up         ARISTA05T2      10.0.0.9
  Loopback0            10.0.1.13/32                                      up/up
  eth0                10.193.0.173/16                    mgmt          up/up


<code>sudo config interface  shutdown Ethernet8 </code>
<code>sudo config interface  shutdown Ethernet8 </code>
Line 61: Line 73:


<code>sudo config interface  startup  Ethernet8 </code>
<code>sudo config interface  startup  Ethernet8 </code>


=== Leaf/spine configuration ===
=== Leaf/spine configuration ===
Line 71: Line 84:


on S5232F-ON (Spine)
on S5232F-ON (Spine)
  Interface    Oper    Admin           Alias    Description
Interface    Oper    Admin   Alias    Description
  -----------  ------  -------  ---------------  -------------
-----------  ------  -------  -------  -------------
  Ethernet0   down       up   hundredGigE1/1            N/A
Ethernet0     down    down  Eth1/1            N/A
   Ethernet4   down       up   hundredGigE1/2           N/A
Ethernet4    down    down   Eth1/2            N/A
   Ethernet8   down       up   hundredGigE1/3           N/A
Ethernet8    down    down  Eth1/3            N/A
Ethernet12   down     down   Eth1/4           N/A
Ethernet16    down    down   Eth1/5            N/A
Ethernet20   down     down   Eth1/6           N/A
   -
   -
   -
   -
  Ethernet120    up      up hundredGigE1/31            N/A
Ethernet120    down    down Eth1/31            N/A
  Ethernet124    up      up hundredGigE1/32            N/A
Ethernet124    down    down Eth1/32            N/A
  Ethernet128  down      up      tenGigE1/33            N/A
Ethernet128   down    down Eth1/33            N/A
  Ethernet129  down      up      tenGigE1/34            N/A
Ethernet129   down    down Eth1/34            N/A
    
    
on both spines switches, we will use interfaces Ethernet120 on Sonic side which is interface E1/31 on Dell switch and Ethernet124 which is E1/32
on both spines switches, we will use interfaces Ethernet120 on Sonic side which is interface E1/31 on Dell switch and Ethernet124 which is E1/32
Line 88: Line 104:
   
   
<code>leaf1# show interface description</code>
<code>leaf1# show interface description</code>
   Interface    Oper    Admin                 Alias    Description
   Interface    Oper    Admin   Alias    Description
  -----------  ------  -------  --------------------  -------------
-----------  ------  -------  -------  -------------
   Ethernet0    down       up   twentyfiveGigE1/1/1            N/A
   Ethernet0    down     down   Eth1/1            N/A
   Ethernet1    down       up   twentyfiveGigE1/1/2            N/A
   Ethernet1    down     down   Eth1/2            N/A
   Ethernet2    down       up   twentyfiveGigE1/1/3           N/A
   Ethernet2    down     down   Eth1/3            N/A
  Ethernet3    down    down  Eth1/4           N/A
   -
   -
   -
   -
   Ethernet54     up      up      hundredGigE1/55           N/A
   Ethernet72  down     down  Eth1/55           N/A
   Ethernet55     up      up      hundredGigE1/56           N/A
   Ethernet76  down     down  Eth1/56           N/A
On the leaf's we will use interfaces Ethernet54 and Ethernet55 which are E1/55 and E1/56 on the Dell switch
On the leaf's we will use interfaces Ethernet72 and Ethernet76 which are E1/55 and E1/56 on the Dell switch


Now that we have the interface mapping from Sonic to Dell it is time to setup IP address on each of those interfaces. Below is the diagram showing the IP's and network that we will use for the leaf/spine setup.
Now that we have the interface mapping from Sonic to Dell it is time to setup IP address on each of those interfaces. Below is the diagram showing the IP's and network that we will use for the leaf/spine setup.
Line 104: Line 121:


==== Interface configuration ====
==== Interface configuration ====
By default the Sonic OS comes with all the interfaces already preconfigured with some IP's.You can check that by looking at the "config_db.json"
<code> admin@dell-spine1:/etc/sonic$ vi config_db.json</code>
"Ethernet120|10.0.0.60/31": {},
"Ethernet124|10.0.0.62/31": {},
"Ethernet128|10.0.0.64/31": {},
"Ethernet129|10.0.0.66/31": {},
"Ethernet12|10.0.0.6/31": {},
"Ethernet16|10.0.0.8/31": {},
We see the two(2) interfaces Ethernet120 and 124  that we are going to use have already an IP address. When setting up the interface IP we need to delete as well the existing IP address.


'''spine1'''
'''spine1'''
  sudo config interface ip add  Ethernet120 172.16.0.9/30
  sonic-cli
  sudo config interface ip remove Ethernet120 10.0.0.60/31
configure terminal
  sudo config interface ip add  Ethernet124 172.16.0.1/30
interface Ethernet120
  sudo config interface ip remove  Ethernet120 10.0.0.62/31
description dell-leaf2
  Sudo config save -y
no shutdown
ip address 172.16.1.9/30
  interface Ethernet124
description dell-leaf1
  no shutdown
  ip address 172.16.1.1/30
  end
  write memory


'''spine2'''
'''spine2'''
  sudo config interface ip add Ethernet124 172.16.0.5/30
  sonic-cli
  sudo config interface ip add Ethernet120 172.16.0.13/30
configure terminal
  sudo config interface ip remove Ethernet120 10.0.0.60/30
interface Ethernet120
  sudo config interface ip remove Ethernet124 10.0.0.62/30       
description dell-leaf2
Sudo config save -y
no shutdown
ip address 172.16.1.14/30
  interface Ethernet124
description dell-leaf1
no shutdown
ip address 172.16.1.6/30
  end
  write memory
'''leaf1'''
'''leaf1'''
  sudo config interface ip add  Ethernet54 172.16.0.6/30
  sonic-cli
  sudo config interface ip remove Ethernet54 10.0.0.108/31
configure terminal
  sudo config interface ip add  Ethernet55 172.16.0.2/30
interface Ethernet72
sudo config interface ip remove  Ethernet55 10.0.0.110/31
description dell-spine2
Sudo config save -y
no shutdown
ip address 172.16.1.6/30
  interface Ethernet76
description dell-spine1
  no shutdown
  ip address 172.16.1.2/30


  '''leaf2'''
  '''leaf2'''
  sudo config interface ip add  Ethernet54 172.16.0.14/30
  sonic-cli
  sudo config interface ip remove Ethernet54 10.0.0.108/31
configure terminal
  sudo config interface ip add  Ethernet55 172.16.0.10/30
interface Ethernet72
sudo config interface ip remove  Ethernet55 10.0.0.110/31
description dell-spine2
Sudo config save -y
no shutdown
ip address 172.16.1.14/30
  interface Ethernet76
description dell-spine1
  no shutdown
  ip address 172.16.1.10/30


Check connectivity between spine1 and leaf1
Check connectivity between spine1 and leaf1
  admin@dell-spine1:/etc/sonic$ ping 172.16.0.2
  dell-spine1# ping 172.16.1.2
  PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
  PING 172.16.1.2 (172.16.1.2) 56(84) bytes of data.
  64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=0.246 ms
  64 bytes from 172.16.1.2: icmp_seq=1 ttl=64 time=0.348 ms
  64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.235 ms
64 bytes from 172.16.1.2: icmp_seq=2 ttl=64 time=0.274 ms
64 bytes from 172.16.1.2: icmp_seq=3 ttl=64 time=0.268 ms
 
dell-leaf1# ping 172.16.1.1
PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.
  64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=0.339 ms
64 bytes from 172.16.1.1: icmp_seq=2 ttl=64 time=0.303 ms


  admin@leaf1:/etc/sonic$ ping 172.16.0.1
  dell-spine2# ping 172.16.1.14
  PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
  PING 172.16.1.14 (172.16.1.14) 56(84) bytes of data.
  64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=0.268 ms
  64 bytes from 172.16.1.14: icmp_seq=1 ttl=64 time=0.356 ms
  64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.241 ms
  64 bytes from 172.16.1.14: icmp_seq=2 ttl=64 time=0.296 ms  
Check connectivity between spine2/leaf1/leaf2


admin@dell-spine2:~$ ping 172.16.0.14
PING 172.16.0.14 (172.16.0.14) 56(84) bytes of data.
64 bytes from 172.16.0.14: icmp_seq=1 ttl=64 time=0.429 ms
64 bytes from 172.16.0.14: icmp_seq=2 ttl=64 time=0.248 ms
admin@dell-spine2:~$ ping 172.16.0.6
PING 172.16.0.6 (172.16.0.6) 56(84) bytes of data.
bytes from 172.16.0.6: icmp_seq=1 ttl=64 time=0.264 ms
Now that we have connectivity, the next step is to setup BGP
Now that we have connectivity, the next step is to setup BGP


====BGP configuration====
====BGP configuration====
There are two(2) ways to configure BGP


- using command line (shell for FRR daemons). I am still testing this method
'''spine1'''
sonic-cli
configure terminal
router bgp 65030
router-id 10.0.1.13
address-family ipv4 unicast
  redistribute connected
neighbor 172.16.1.10
  remote-as 65033
  address-family ipv4 unicast
  activate
neighbor 172.16.1.2
  remote-as 65032
  address-family ipv4 unicast
  activate
end
write memory


<code>sudo vtysh</code>
'''spine2'''
 
sonic-cli
- Make the changes directly from the config_db.json file. If you make the changes from this file you need to run
configure terminal
router bgp 65030
router-id 10.0.1.14
address-family ipv4 unicast
  redistribute connected
neighbor 172.16.1.14
  remote-as 65033
  address-family ipv4 unicast
  activate
neighbor 172.16.1.6
  remote-as 65032
  address-family ipv4 unicast
  activate
  end
  write memory


<code> sudo config reload -y </code> to save the changes to the Database.
'''leaf 1'''
router bgp 65032
router-id 10.0.1.24
address-family ipv4 unicast
  redistribute connected
neighbor 172.16.1.1
  remote-as 65030
  address-family ipv4 unicast
  activate
neighbor 172.16.1.5
  remote-as 65030
  address-family ipv4 unicast
  activate
end
write memory


Sonic comes with some BGP neighbor's settings preconfigured. We have the option to delete those settings and add new once or just add new once. Note: The preconfigured settings are not active.See below
'''leaf 2'''
  "BGP_NEIGHBOR": {
  router bgp 65033
    "10.0.0.1": {
router-id 10.0.1.25
        "asn": "65200",
  address-family ipv4 unicast
        "holdtime": "180",
  redistribute connected
        "keepalive": "60",
  neighbor 172.16.1.9
        "local_addr": "10.0.0.0",
  remote-as 65030
        "name": "ARISTA01T2",
  address-family ipv4 unicast
        "nhopself": "0",
  activate
        "rrclient": "0"
  neighbor 172.16.1.13
    },
   remote-as 65030
    "10.0.0.3": {
  address-family ipv4 unicast
    "asn": "65200",
  activate
    "holdtime": "180",
end
    "keepalive": "60",
write memory
    "local_addr": "10.0.0.2",
    "name": "ARISTA02T2",
    "nhopself": "0",
    "rrclient": "0"
For our testing, we are going to replace the first two(2) preconfigured neighbors with our settings. First we need to setup the Loopback0 interface IP since we will use this in the BGP configuration for the router identifier (router-id).
like for the Ethernetx configuration, the Loopback0 interface also comes with a default IP address of 10.1.0.1/32
sudo config interface ip add Loopback0 10.0.1.13/32 (spine1)
  sudo config interface ip remove Loopback0 10.1.0.1/32
sudo config save -y
 
  sudo config interface ip add Loopback0 10.0.1.14/32 (spine2)
sudo config interface ip remove Loopback0 10.1.0.1/32
sudo config save -y
 
sudo config interface ip add Loopback0 10.0.1.24/32 (leaf1)
sudo config interface ip remove Loopback0 10.1.0.1/32
sudo config save -y
 
  sudo config interface ip add Loopback0 10.0.1.25/32 (leaf2)
sudo config interface ip remove Loopback0 10.1.0.1/32
sudo config save -y
 
'''spine1'''
"BGP_NEIGHBOR": {
    "172.16.0.2": {
        "asn": "65032",
        "holdtime": "180",
        "keepalive": "60",
        "local_addr": "10.0.1.13",
        "name": "leaf1",
        "nhopself": "0",
        "rrclient": "0"
    },
    "172.16.0.10": {
    "asn": "65033",
    "holdtime": "180",
    "keepalive": "60",
    "local_addr": "10.0.1.13",
    "name": "leaf2",
    "nhopself": "0",
    "rrclient": "0"
Under "DEVICE_METADATA" change "bgp_asn": "65100" to spine1 asn number
    
"DEVICE_METADATA": {
    "localhost": {
    "bgp_asn": "65030",
Close the file and run <code> sudo config reload -y</code>
 
Repeat the same steps for spine2, leaf1 and leaf2 by replacing the "IP addresses" and "asn"


''' verification'''
''' verification'''


<code>show ip bgp summary</code>
<code>show bgp ipv4 unicast summary</code>


'''spine1'''
'''spine1'''
  Neighbhor      V     AS   MsgRcvd   MsgSent   TblVer    InQ    OutQ  Up/Down     State/PfxRcd NeighborName
  BGP router identifier 10.0.1.13, local AS number 65030
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
Neighbor        V   AS     MsgRcvd   MsgSent   InQ     OutQ   Up/Down         State/PfxRcd
  172.16.0.2     4 65032       1566       1569        0     0      0  1d01h57m                3 leaf1
  172.16.1.10     4   65033  142       141       0       0      02:13:29        3
  172.16.0.10    4 65033       1534       1543        0     0      0  1d01h27m                2  leaf2
  172.16.1.2      4   65032  152       163       0       0      02:23:07        3
  Total number of neighbors 2
  Total number of neighbors 2
Total number of neighbors established 2
   
   
'''spine2'''
'''spine2'''
  Neighbhor      V     AS   MsgRcvd   MsgSent   TblVer    InQ    OutQ  Up/Down     State/PfxRcd NeighborName
  BGP router identifier 10.0.1.14, local AS number 65030
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
Neighbor        V   AS     MsgRcvd   MsgSent   InQ     OutQ   Up/Down         State/PfxRcd
  172.16.0.6     4 65032       1548       1548        0     0      0  1d01h40m                2  leaf1
  172.16.1.14     4   65033  145       144       0       0      02:16:31        3
  172.16.0.14    4 65033       1543       1544        0     0      0  1d01h28m                3 leaf2
  172.16.1.6      4   65032  160       159       0       0      02:31:48        3
  Total number of neighbors 2
  Total number of neighbors 2
Total number of neighbors established 2


'''leaf1'''
'''leaf1'''
  Neighbhor      V     AS   MsgRcvd   MsgSent   TblVer    InQ    OutQ  Up/Down     State/PfxRcd NeighborName
  BGP router identifier 10.0.1.24, local AS number 65032
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
Neighbor      V   AS     MsgRcvd   MsgSent   InQ     OutQ   Up/Down         State/PfxRcd
  172.16.0.1    4 65030      1562       1561        0     0      0  1d01h53m                2  Spine1
  172.16.1.1    4   65030   156       159       0       0      02:27:13        5
  172.16.0.5    4 65031       1543       1647        0     0      0  1d01h35m                2  spine2
  172.16.1.5    4   65030  161       164       0       0      02:32:49        5
  Total number of neighbors 2
  Total number of neighbors 2
Total number of neighbors established 2


'''leaf2'''
'''leaf2'''
  Neighbhor      V     AS   MsgRcvd   MsgSent   TblVer    InQ    OutQ  Up/Down     State/PfxRcd  NeighborName
  BGP router identifier 10.0.1.25, local AS number 65033
  ----------- --- -----  ---------  --------- --------  -----  ------  --------- -------------- --------------
Neighbor        V   AS     MsgRcvd   MsgSent   InQ     OutQ   Up/Down         State/PfxRcd
  172.16.0.9    4 65030      1539      1537        0     0       1d01h30m   3              spine1
172.16.1.13    4  65030  151      154      0      0      02:22:01        5
  172.16.0.13    4 65031      1538      1537        0     0       1d01h30m  3               spine2
172.16.1.9      4  65030  151      154      0      0      02:22:04        5
  Total number of neighbors 2
  Total number of neighbors established 2
 
=== EVPN/VXLAN Configuration ===
 
====Spine====
On both spine's the only thing needed to be configured is EVPN and the configuration is the same some both.
 
'''Spine1'''
 
router bgp 65030
!
address-family l2vpn evpn
!
neighbor 172.16.1.10
  !
  address-family l2vpn evpn
  activate
!
neighbor 172.16.1.2
  !
  address-family l2vpn evpn
  activate
 
'''Verification'''
 
dell-spine1# show bgp l2vpn evpn summary
BGP router identifier 10.0.1.13, local AS number 65030
Neighbor        V  AS      MsgRcvd  MsgSent  InQ    OutQ    Up/Down        State/PfxRcd
172.16.1.10    4  65033  9612      9605      0      0      00:00:35        0
172.16.1.2      4  65032  9627      9622      0      0      00:00:33        0
 
'''Spine2'''
 
router bgp 65030
!
address-family l2vpn evpn
!
neighbor 172.16.1.14
  !
  address-family l2vpn evpn
  activate
!
neighbor 172.16.1.6
  !
  address-family l2vpn evpn
  activate
 
'''Verification'''
 
dell-spine2# show bgp l2vpn evpn summary
BGP router identifier 10.0.1.14, local AS number 65030
Neighbor        V  AS      MsgRcvd  MsgSent  InQ    OutQ    Up/Down        State/PfxRcd
172.16.1.14    4  65033  9612      9605      0      0      00:00:35        0
172.16.1.6      4  65032  9627      9622      0      0      00:00:33        0
 
====Leaf====
 
'''leaf1'''
interface Loopback 1
  description "VXLAN source IP"
  ip address 10.10.10.1/32
 
  interface vxlan vtep1
  source-ip 10.10.10.1
 
  router bgp 65032
  address-family l2vpn evpn
  advertise-all-vni
  !
  neighbor 172.16.1.1
  !
  address-family l2vpn evpn
    activate
  !
  neighbor 172.16.1.5
  !
  address-family l2vpn evpn
    activate
ip vrf Vrf_codfw
  ip anycast-mac-address 00:00:00:10:10:10
router bgp 65032 vrf Vrf_codfw
router-id 10.0.1.24
!
address-family ipv4 unicast
  redistribute connected
!
address-family l2vpn evpn
  advertise ipv4 unicast
!
 
interface Vlan4000
  description "IRB VLAN"
  ip vrf forwarding Vrf_codfw
  ipv6 enable
!
interface Vlan2004
  description private1-e-codfw
  ip vrf forwarding Vrf_codfw
  ip anycast-address 10.192.64.254/22
!
interface vxlan vtep1
  map vni 102004 vlan 2004
  map vni 404000 vlan 4000
  map vni 404000 vrf Vrf_codfw
!
interface Ethernet0
  description test_srv1
  no shutdown
  switchport access Vlan 2004
 
'''Verification'''
dell-leaf1# show vxlan tunnel
Name                SIP              DIP                source      operstatus
  =======            ======            ======              ======      ========
 
The vxlan tunnel is not formed yet, because the switch hasn't learned any routes from the end devices (servers)
dell-leaf1# show vxlan vlanvnimap
VLAN      VNI
======    =====
Vlan2004  102004
Vlan4000  404000
Total count :    2
dell-leaf1# show vxlan vrfvnimap
VRF      VNI
======    =====
Vrf_codfw  404000
Total count :    1
 
'''leaf 2'''
interface Loopback 1
  description "VXLAN source IP"
  ip address 10.10.10.2/32
 
interface vxlan vtep1
  source-ip 10.10.10.2
 
router bgp 65033
  address-family l2vpn evpn
  advertise-all-vni
  !
  neighbor 172.16.1.9
  !
  address-family l2vpn evpn
    activate
  !
  neighbor 172.16.1.13
  !
  address-family l2vpn evpn
    activate
ip vrf Vrf_codfw
  ip anycast-mac-address 00:00:00:10:10:10
router bgp 65033 vrf Vrf_codfw
router-id 10.0.1.25
!
address-family ipv4 unicast
  redistribute connected
!
address-family l2vpn evpn
  advertise ipv4 unicast
!
 
interface Vlan4000
  description "IRB VLAN"
  ip vrf forwarding Vrf_codfw
  ipv6 enable
!
interface Vlan2004
  description private1-e-codfw
  ip vrf forwarding Vrf_codfw
  ip anycast-address 10.192.64.254/22
!
interface vxlan vtep1
  map vni 102004 vlan 2004
  map vni 404000 vlan 4000
  map vni 404000 vrf Vrf_codfw
!
  interface Ethernet0
  description test_srv2
  no shutdown
  switchport access Vlan 2004
 
'''Verification'''
dell-leaf1# show vxlan tunnel
Name                SIP              DIP                source      operstatus
=======            ======            ======              ======      ========
 
The vxlan tunnel is not formed yet, because the switch hasn't learned any routes from the end devices (servers)
dell-leaf2# show vxlan vlanvnimap
VLAN      VNI
======    =====
Vlan2004  102004
Vlan4000  404000
Total count :    2
dell-leaf1# show vxlan vrfvnimap
VRF      VNI
======    =====
Vrf_codfw  404000
Total count :    1
 
== End devices connectivity ==
 
I setup up 2 servers, one with IP address 10.192.64.10/22 connected to leaf 1 and the second with IP address 10.192.64.20/22 connected to leaf2. After running a ping test from server one to server 2. I checked the vxlan tunnel on both leaf's switches.
 
dell-leaf1# show vxlan tunnel
Name                SIP              DIP                source      operstatus
=======            ======            ======              ======      ========
EVPN_10.10.10.2    10.10.10.1        10.10.10.2          EVPN        oper_up
dell-leaf2# show vxlan tunnel
Name                SIP              DIP                source      operstatus
=======            ======            ======              ======      ========
  EVPN_10.10.10.1    10.10.10.2        10.10.10.1          EVPN        oper_up
 
=== Some routes informations ===
 
''' type 5 routes '''
 
dell-leaf2# show bgp l2vpn evpn route type prefix
BGP table version is 19, local router ID is 10.0.1.25
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]
  EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
  EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
  EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
  Network          Next Hop            Metric LocPrf Weight Path
                    Extended Community
  Route Distinguisher: 10.0.1.24:5096
  *    [5]:[0]:[22]:[10.192.64.0]
                    10.10.10.1                                    0 65030 65032 ?
                    RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
  *>   [5]:[0]:[22]:[10.192.64.0]
                    10.10.10.1                                    0 65030 65032 ?
                    RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
  Route Distinguisher: 10.0.1.25:5096
  *>  [5]:[0]:[22]:[10.192.64.0]
                    10.10.10.2              0                 32768 ?
                    ET:8 RT:65033:40400 Rmac:3c:2c:30:4c:81:83
  Displayed 2 prefixes (3 paths) (of requested type)
=== Test validation ===
 
Network functionality and failover testing is documented on the [[VXLAN-EVPN Network Testing - Sonic on Dell switches]] page.


=== Known limitations ===
=== Known limitations ===

Latest revision as of 16:15, 29 March 2022

Configuration

Currently tested on the April 2021 release.

Manual

All configuration change done with sudo config need at some point a sudo config save to make them permanent (eg. device reboot)

Hostname

sudo config hostname dell-spine1

Management

SONiC supports having the management interface in a dedicated VRF, so the default management route (to mr1) doesn't risk impacting production traffic.

sudo config vrf add mgmt

sudo config interface ip add eth0 10.x.x.x/16 10.x.0.1

show management_interface address

https://github.com/Azure/SONiC/blob/master/doc/mgmt/sonic_stretch_management_vrf_design.md

https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md#configuring-management-interface

Note: for the Enterprise Sonic the only commands needed are

ip address 10.193.0.x/16 gwaddr 10.193.0.1

ip vrf mgmt

SNMP

Have SNMP listen on its mgmt IP

sudo config snmpagentaddress add 10.x.x.x -v mgmt -p 161

In the tested release, sudo config snmp is not a valid configuration keyword.

Workaround to set the SNMP community:

sudo vim /etc/sonic/snmp.yml

sudo systemctl restart snmp

https://github.com/Azure/SONiC/wiki/How-to-Check-SNMP-Configuration

Note: for the Enterprise Sonic the only command needed is snmp-server community <YourCommunity>

admin@dell-leaf2:~$ sonic-cli
dell-leaf2# configure terminal
dell-leaf2(config)# snmp-server community <yourcommunity>

NTP

sudo config ntp add <server IP>

Some commands

show ip interface

-----------------------------------------------------------------------------------------------
Interface            IP address/mask                    VRF            Admin/Oper     Flags
-----------------------------------------------------------------------------------------------
Ethernet120          172.16.1.9/30                                     up/up
Ethernet124          172.16.1.1/30                                     up/up
Loopback0            10.0.1.13/32                                      up/up
eth0                 10.193.0.173/16                    mgmt           up/up

sudo config interface shutdown Ethernet8

Interface    Master    IPv4 address/mask    Admin/Oper    BGP Neighbor    Neighbor IP
-----------  --------  -------------------  ------------  --------------  -------------
Ethernet4              10.0.0.2/31          up/up         ARISTA02T2      10.0.0.3
Ethernet8              10.0.0.4/31          down/down     ARISTA03T2      10.0.0.5
Ethernet12             10.0.0.6/31          up/up         ARISTA04T2      10.0.0.7

sudo config interface startup Ethernet8


Leaf/spine configuration

We are going to use the diagram below to setup the leaf/spine. Each lead switch has 1 connection to each spine switch. There is no connection between the spines. The first setup is to identify the interfaces on each switch.

File:Sonic leaf spine1.png

Interface mapping

show interfaces description

on S5232F-ON (Spine)

Interface    Oper    Admin    Alias    Description
-----------  ------  -------  -------  -------------
Ethernet0     down     down   Eth1/1            N/A
Ethernet4     down     down   Eth1/2            N/A
Ethernet8     down     down   Eth1/3            N/A
Ethernet12    down     down   Eth1/4            N/A
Ethernet16    down     down   Eth1/5            N/A
Ethernet20    down     down   Eth1/6            N/A
 -
 -
Ethernet120    down    down  Eth1/31            N/A
Ethernet124    down    down  Eth1/32            N/A
Ethernet128    down    down  Eth1/33            N/A
Ethernet129    down    down  Eth1/34            N/A
 

on both spines switches, we will use interfaces Ethernet120 on Sonic side which is interface E1/31 on Dell switch and Ethernet124 which is E1/32

On S5248F-ON (leaf)

leaf1# show interface description

 Interface    Oper    Admin    Alias    Description
-----------  ------  -------  -------  -------------
 Ethernet0    down     down   Eth1/1            N/A
 Ethernet1    down     down   Eth1/2            N/A
 Ethernet2    down     down   Eth1/3            N/A
 Ethernet3    down     down   Eth1/4            N/A
 -
 -
 Ethernet72   down     down   Eth1/55           N/A
 Ethernet76   down     down   Eth1/56           N/A

On the leaf's we will use interfaces Ethernet72 and Ethernet76 which are E1/55 and E1/56 on the Dell switch

Now that we have the interface mapping from Sonic to Dell it is time to setup IP address on each of those interfaces. Below is the diagram showing the IP's and network that we will use for the leaf/spine setup.

File:Sonic leaf spine2.png

Interface configuration

spine1

sonic-cli
configure terminal
interface Ethernet120
description dell-leaf2
no shutdown
ip address 172.16.1.9/30
interface Ethernet124
description dell-leaf1
no shutdown
ip address 172.16.1.1/30
end 
write memory

spine2

sonic-cli
configure terminal
interface Ethernet120
description dell-leaf2
no shutdown
ip address 172.16.1.14/30
interface Ethernet124
description dell-leaf1
no shutdown
ip address 172.16.1.6/30
end 
write memory

leaf1

sonic-cli
configure terminal
interface Ethernet72
description dell-spine2
no shutdown
ip address 172.16.1.6/30
interface Ethernet76
description dell-spine1
no shutdown
ip address 172.16.1.2/30
leaf2
sonic-cli
configure terminal
interface Ethernet72
description dell-spine2
no shutdown
ip address 172.16.1.14/30
interface Ethernet76
description dell-spine1
no shutdown
ip address 172.16.1.10/30

Check connectivity between spine1 and leaf1

dell-spine1# ping 172.16.1.2
PING 172.16.1.2 (172.16.1.2) 56(84) bytes of data.
64 bytes from 172.16.1.2: icmp_seq=1 ttl=64 time=0.348 ms
64 bytes from 172.16.1.2: icmp_seq=2 ttl=64 time=0.274 ms
64 bytes from 172.16.1.2: icmp_seq=3 ttl=64 time=0.268 ms
dell-leaf1# ping 172.16.1.1
PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=0.339 ms
64 bytes from 172.16.1.1: icmp_seq=2 ttl=64 time=0.303 ms
dell-spine2# ping 172.16.1.14
PING 172.16.1.14 (172.16.1.14) 56(84) bytes of data.
64 bytes from 172.16.1.14: icmp_seq=1 ttl=64 time=0.356 ms
64 bytes from 172.16.1.14: icmp_seq=2 ttl=64 time=0.296 ms 

Now that we have connectivity, the next step is to setup BGP

BGP configuration

spine1

sonic-cli
configure terminal
router bgp 65030
router-id 10.0.1.13
address-family ipv4 unicast
 redistribute connected
neighbor 172.16.1.10
 remote-as 65033
 address-family ipv4 unicast
  activate
neighbor 172.16.1.2
 remote-as 65032
 address-family ipv4 unicast
  activate
end
write memory

spine2

sonic-cli
configure terminal
router bgp 65030
router-id 10.0.1.14
address-family ipv4 unicast
 redistribute connected
neighbor 172.16.1.14
 remote-as 65033
 address-family ipv4 unicast
  activate
neighbor 172.16.1.6
 remote-as 65032
 address-family ipv4 unicast
  activate 
 end
 write memory

leaf 1

router bgp 65032
router-id 10.0.1.24
address-family ipv4 unicast
 redistribute connected
neighbor 172.16.1.1
 remote-as 65030
 address-family ipv4 unicast
  activate
neighbor 172.16.1.5
 remote-as 65030
 address-family ipv4 unicast
  activate
end
write memory

leaf 2

router bgp 65033
router-id 10.0.1.25
address-family ipv4 unicast
 redistribute connected
neighbor 172.16.1.9
 remote-as 65030
 address-family ipv4 unicast
  activate
neighbor 172.16.1.13
 remote-as 65030
 address-family ipv4 unicast
  activate
end
write memory

verification

show bgp ipv4 unicast summary

spine1

BGP router identifier 10.0.1.13, local AS number 65030
Neighbor        V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd
172.16.1.10     4   65033   142       141       0       0       02:13:29        3
172.16.1.2      4   65032   152       163       0       0       02:23:07        3
Total number of neighbors 2
Total number of neighbors established 2

spine2

BGP router identifier 10.0.1.14, local AS number 65030
Neighbor        V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd
172.16.1.14     4   65033   145       144       0       0       02:16:31        3
172.16.1.6      4   65032   160       159       0       0       02:31:48        3
Total number of neighbors 2
Total number of neighbors established 2

leaf1

BGP router identifier 10.0.1.24, local AS number 65032
Neighbor       V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd
172.16.1.1     4   65030   156       159       0       0       02:27:13        5
172.16.1.5     4   65030   161       164       0       0       02:32:49        5
Total number of neighbors 2
Total number of neighbors established 2

leaf2

BGP router identifier 10.0.1.25, local AS number 65033
Neighbor        V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd
172.16.1.13     4   65030   151       154       0       0       02:22:01        5
172.16.1.9      4   65030   151       154       0       0       02:22:04        5
Total number of neighbors 2
Total number of neighbors established 2

EVPN/VXLAN Configuration

Spine

On both spine's the only thing needed to be configured is EVPN and the configuration is the same some both.

Spine1

router bgp 65030
!
address-family l2vpn evpn
!
neighbor 172.16.1.10
 !
 address-family l2vpn evpn
  activate
!
neighbor 172.16.1.2
 !
 address-family l2vpn evpn
  activate

Verification

dell-spine1# show bgp l2vpn evpn summary
BGP router identifier 10.0.1.13, local AS number 65030
Neighbor        V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd
172.16.1.10     4   65033   9612      9605      0       0       00:00:35        0
172.16.1.2      4   65032   9627      9622      0       0       00:00:33        0

Spine2

router bgp 65030
!
address-family l2vpn evpn
!
neighbor 172.16.1.14
 !
 address-family l2vpn evpn
  activate
!
neighbor 172.16.1.6
 !
 address-family l2vpn evpn
  activate

Verification

dell-spine2# show bgp l2vpn evpn summary
BGP router identifier 10.0.1.14, local AS number 65030
Neighbor        V   AS      MsgRcvd   MsgSent   InQ     OutQ    Up/Down         State/PfxRcd
172.16.1.14     4   65033   9612      9605      0       0       00:00:35        0
172.16.1.6      4   65032   9627      9622      0       0       00:00:33        0

Leaf

leaf1

interface Loopback 1
 description "VXLAN source IP"
 ip address 10.10.10.1/32
interface vxlan vtep1
 source-ip 10.10.10.1
router bgp 65032
 address-family l2vpn evpn
  advertise-all-vni
 !
 neighbor 172.16.1.1
  !
  address-family l2vpn evpn
   activate
 !
 neighbor 172.16.1.5
  !
  address-family l2vpn evpn
   activate
ip vrf Vrf_codfw
ip anycast-mac-address 00:00:00:10:10:10

router bgp 65032 vrf Vrf_codfw
router-id 10.0.1.24
!
address-family ipv4 unicast
 redistribute connected
!
address-family l2vpn evpn
 advertise ipv4 unicast
!
interface Vlan4000
 description "IRB VLAN"
 ip vrf forwarding Vrf_codfw
 ipv6 enable
!
interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.64.254/22
!
interface vxlan vtep1
 map vni 102004 vlan 2004
 map vni 404000 vlan 4000
 map vni 404000 vrf Vrf_codfw
!
interface Ethernet0
 description test_srv1
 no shutdown
 switchport access Vlan 2004

Verification

dell-leaf1# show vxlan tunnel
Name                SIP               DIP                 source      operstatus
=======             ======            ======              ======      ========

The vxlan tunnel is not formed yet, because the switch hasn't learned any routes from the end devices (servers)

dell-leaf1# show vxlan vlanvnimap
VLAN      VNI
======    =====
Vlan2004  102004
Vlan4000  404000
Total count :    2

dell-leaf1# show vxlan vrfvnimap
VRF       VNI
======    =====
Vrf_codfw  404000
Total count :    1

leaf 2

interface Loopback 1
 description "VXLAN source IP"
 ip address 10.10.10.2/32
interface vxlan vtep1
 source-ip 10.10.10.2
router bgp 65033
 address-family l2vpn evpn
  advertise-all-vni
 !
 neighbor 172.16.1.9
  !
  address-family l2vpn evpn
   activate
 !
 neighbor 172.16.1.13
  !
  address-family l2vpn evpn
   activate
ip vrf Vrf_codfw
ip anycast-mac-address 00:00:00:10:10:10

router bgp 65033 vrf Vrf_codfw
router-id 10.0.1.25
!
address-family ipv4 unicast
 redistribute connected
!
address-family l2vpn evpn
 advertise ipv4 unicast
!
interface Vlan4000
 description "IRB VLAN"
 ip vrf forwarding Vrf_codfw
 ipv6 enable
!
interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.64.254/22
!
interface vxlan vtep1
 map vni 102004 vlan 2004
 map vni 404000 vlan 4000
 map vni 404000 vrf Vrf_codfw
!
interface Ethernet0
 description test_srv2
 no shutdown
 switchport access Vlan 2004

Verification

dell-leaf1# show vxlan tunnel
Name                SIP               DIP                 source      operstatus
=======             ======            ======              ======      ========

The vxlan tunnel is not formed yet, because the switch hasn't learned any routes from the end devices (servers)

dell-leaf2# show vxlan vlanvnimap
VLAN      VNI
======    =====
Vlan2004  102004
Vlan4000  404000
Total count :    2

dell-leaf1# show vxlan vrfvnimap
VRF       VNI
======    =====
Vrf_codfw  404000
Total count :    1

End devices connectivity

I setup up 2 servers, one with IP address 10.192.64.10/22 connected to leaf 1 and the second with IP address 10.192.64.20/22 connected to leaf2. After running a ping test from server one to server 2. I checked the vxlan tunnel on both leaf's switches.

dell-leaf1# show vxlan tunnel
Name                SIP               DIP                 source      operstatus
=======             ======            ======              ======      ========
EVPN_10.10.10.2     10.10.10.1        10.10.10.2          EVPN        oper_up

dell-leaf2# show vxlan tunnel
Name                SIP               DIP                 source      operstatus
=======             ======            ======              ======      ========
EVPN_10.10.10.1     10.10.10.2        10.10.10.1          EVPN        oper_up

Some routes informations

type 5 routes

dell-leaf2# show bgp l2vpn evpn route type prefix
BGP table version is 19, local router ID is 10.0.1.25
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
  Network          Next Hop            Metric LocPrf Weight Path
                   Extended Community
Route Distinguisher: 10.0.1.24:5096
*    [5]:[0]:[22]:[10.192.64.0]
                   10.10.10.1                                     0 65030 65032 ?
                   RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
*>   [5]:[0]:[22]:[10.192.64.0]
                   10.10.10.1                                     0 65030 65032 ?
                   RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
Route Distinguisher: 10.0.1.25:5096
*>   [5]:[0]:[22]:[10.192.64.0]
                   10.10.10.2               0                 32768 ?
                   ET:8 RT:65033:40400 Rmac:3c:2c:30:4c:81:83
Displayed 2 prefixes (3 paths) (of requested type)

Test validation

Network functionality and failover testing is documented on the VXLAN-EVPN Network Testing - Sonic on Dell switches page.

Known limitations

  • It is not possible to configure a DNS resolver, all configuration needs to be done using IPs
  • It is not possible to change the syslog facility
  • cgexec -g l3mdev:mgmt is supposed to run commands from the mgmt VRF, but fails with "cgroup change of group failed"
    • "ip vrf exec <name> <command>" does seem to work for this. (name being the vrf name, "mgmt" in this case.)