You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
SONiC: Difference between revisions
imported>Papaul |
imported>Cathal Mooney |
||
| Line 277: | Line 277: | ||
* It is not possible to change the syslog facility | * It is not possible to change the syslog facility | ||
* <code>cgexec -g l3mdev:mgmt</code> is supposed to run commands from the mgmt VRF, but fails with "cgroup change of group failed" | * <code>cgexec -g l3mdev:mgmt</code> is supposed to run commands from the mgmt VRF, but fails with "cgroup change of group failed" | ||
** "ip vrf exec <name> <command>" does seem to work for this. (name being the vrf name, "mgmt" in this case.) | |||
Revision as of 10:04, 1 December 2021
Configuration
Currently tested on the April 2021 release.
Manual
All configuration change done with sudo config need at some point a sudo config save to make them permanent (eg. device reboot)
Hostname
sudo config hostname dell-spine1
Management
SONiC supports having the management interface in a dedicated VRF, so the default management route (to mr1) doesn't risk impacting production traffic.
sudo config vrf add mgmt
sudo config interface ip add eth0 10.x.x.x/16 10.x.0.1
show management_interface address
https://github.com/Azure/SONiC/blob/master/doc/mgmt/sonic_stretch_management_vrf_design.md
SNMP
Have SNMP listen on its mgmt IP
sudo config snmpagentaddress add 10.x.x.x -v mgmt -p 161
In the tested release, sudo config snmp is not a valid configuration keyword.
Workaround to set the SNMP community:
sudo vim /etc/sonic/snmp.yml
sudo systemctl restart snmp
https://github.com/Azure/SONiC/wiki/How-to-Check-SNMP-Configuration
NTP
sudo config ntp add <server IP>
Some commands
show ip interface
Interface Master IPv4 address/mask Admin/Oper BGP Neighbor Neighbor IP ----------- -------- ------------------- ------------ -------------- ------------- Ethernet4 10.0.0.2/31 up/up ARISTA02T2 10.0.0.3 Ethernet8 10.0.0.4/31 up/up ARISTA03T2 10.0.0.5 Ethernet12 10.0.0.6/31 up/up ARISTA04T2 10.0.0.7 Ethernet16 10.0.0.8/31 up/up ARISTA05T2 10.0.0.9
sudo config interface shutdown Ethernet8
Interface Master IPv4 address/mask Admin/Oper BGP Neighbor Neighbor IP ----------- -------- ------------------- ------------ -------------- ------------- Ethernet4 10.0.0.2/31 up/up ARISTA02T2 10.0.0.3 Ethernet8 10.0.0.4/31 down/down ARISTA03T2 10.0.0.5 Ethernet12 10.0.0.6/31 up/up ARISTA04T2 10.0.0.7
sudo config interface startup Ethernet8
Leaf/spine configuration
We are going to use the diagram below to setup the leaf/spine. Each lead switch has 1 connection to each spine switch. There is no connection between the spines. The first setup is to identify the interfaces on each switch.
Interface mapping
show interfaces description
on S5232F-ON (Spine)
Interface Oper Admin Alias Description ----------- ------ ------- --------------- ------------- Ethernet0 down up hundredGigE1/1 N/A Ethernet4 down up hundredGigE1/2 N/A Ethernet8 down up hundredGigE1/3 N/A - - Ethernet120 up up hundredGigE1/31 N/A Ethernet124 up up hundredGigE1/32 N/A Ethernet128 down up tenGigE1/33 N/A Ethernet129 down up tenGigE1/34 N/A
on both spines switches, we will use interfaces Ethernet120 on Sonic side which is interface E1/31 on Dell switch and Ethernet124 which is E1/32
On S5248F-ON (leaf)
leaf1# show interface description
Interface Oper Admin Alias Description ----------- ------ ------- -------------------- ------------- Ethernet0 down up twentyfiveGigE1/1/1 N/A Ethernet1 down up twentyfiveGigE1/1/2 N/A Ethernet2 down up twentyfiveGigE1/1/3 N/A - - Ethernet54 up up hundredGigE1/55 N/A Ethernet55 up up hundredGigE1/56 N/A
On the leaf's we will use interfaces Ethernet54 and Ethernet55 which are E1/55 and E1/56 on the Dell switch
Now that we have the interface mapping from Sonic to Dell it is time to setup IP address on each of those interfaces. Below is the diagram showing the IP's and network that we will use for the leaf/spine setup.
Interface configuration
By default the Sonic OS comes with all the interfaces already preconfigured with some IP's.You can check that by looking at the "config_db.json"
admin@dell-spine1:/etc/sonic$ vi config_db.json
"Ethernet120|10.0.0.60/31": {},
"Ethernet124|10.0.0.62/31": {},
"Ethernet128|10.0.0.64/31": {},
"Ethernet129|10.0.0.66/31": {},
"Ethernet12|10.0.0.6/31": {},
"Ethernet16|10.0.0.8/31": {},
We see the two(2) interfaces Ethernet120 and 124 that we are going to use have already an IP address. When setting up the interface IP we need to delete as well the existing IP address.
spine1
sudo config interface ip add Ethernet120 172.16.0.9/30 sudo config interface ip remove Ethernet120 10.0.0.60/31 sudo config interface ip add Ethernet124 172.16.0.1/30 sudo config interface ip remove Ethernet120 10.0.0.62/31 Sudo config save -y
spine2
sudo config interface ip add Ethernet124 172.16.0.5/30 sudo config interface ip add Ethernet120 172.16.0.13/30 sudo config interface ip remove Ethernet120 10.0.0.60/30 sudo config interface ip remove Ethernet124 10.0.0.62/30 Sudo config save -y
leaf1
sudo config interface ip add Ethernet54 172.16.0.6/30 sudo config interface ip remove Ethernet54 10.0.0.108/31 sudo config interface ip add Ethernet55 172.16.0.2/30 sudo config interface ip remove Ethernet55 10.0.0.110/31 Sudo config save -y
leaf2 sudo config interface ip add Ethernet54 172.16.0.14/30 sudo config interface ip remove Ethernet54 10.0.0.108/31 sudo config interface ip add Ethernet55 172.16.0.10/30 sudo config interface ip remove Ethernet55 10.0.0.110/31 Sudo config save -y
Check connectivity between spine1 and leaf1
admin@dell-spine1:/etc/sonic$ ping 172.16.0.2 PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data. 64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=0.246 ms 64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.235 ms
admin@leaf1:/etc/sonic$ ping 172.16.0.1 PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data. 64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=0.268 ms 64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.241 ms
Check connectivity between spine2/leaf1/leaf2
admin@dell-spine2:~$ ping 172.16.0.14 PING 172.16.0.14 (172.16.0.14) 56(84) bytes of data. 64 bytes from 172.16.0.14: icmp_seq=1 ttl=64 time=0.429 ms 64 bytes from 172.16.0.14: icmp_seq=2 ttl=64 time=0.248 ms admin@dell-spine2:~$ ping 172.16.0.6 PING 172.16.0.6 (172.16.0.6) 56(84) bytes of data. bytes from 172.16.0.6: icmp_seq=1 ttl=64 time=0.264 ms
Now that we have connectivity, the next step is to setup BGP
BGP configuration
There are two(2) ways to configure BGP
- using command line (shell for FRR daemons). I am still testing this method
sudo vtysh
- Make the changes directly from the config_db.json file. If you make the changes from this file you need to run
sudo config reload -y to save the changes to the Database.
Sonic comes with some BGP neighbor's settings preconfigured. We have the option to delete those settings and add new once or just add new once. Note: The preconfigured settings are not active.See below
"BGP_NEIGHBOR": {
"10.0.0.1": {
"asn": "65200",
"holdtime": "180",
"keepalive": "60",
"local_addr": "10.0.0.0",
"name": "ARISTA01T2",
"nhopself": "0",
"rrclient": "0"
},
"10.0.0.3": {
"asn": "65200",
"holdtime": "180",
"keepalive": "60",
"local_addr": "10.0.0.2",
"name": "ARISTA02T2",
"nhopself": "0",
"rrclient": "0"
For our testing, we are going to replace the first two(2) preconfigured neighbors with our settings. First we need to setup the Loopback0 interface IP since we will use this in the BGP configuration for the router identifier (router-id). like for the Ethernetx configuration, the Loopback0 interface also comes with a default IP address of 10.1.0.1/32
sudo config interface ip add Loopback0 10.0.1.13/32 (spine1) sudo config interface ip remove Loopback0 10.1.0.1/32 sudo config save -y
sudo config interface ip add Loopback0 10.0.1.14/32 (spine2) sudo config interface ip remove Loopback0 10.1.0.1/32 sudo config save -y
sudo config interface ip add Loopback0 10.0.1.24/32 (leaf1) sudo config interface ip remove Loopback0 10.1.0.1/32 sudo config save -y
sudo config interface ip add Loopback0 10.0.1.25/32 (leaf2) sudo config interface ip remove Loopback0 10.1.0.1/32 sudo config save -y
spine1
"BGP_NEIGHBOR": {
"172.16.0.2": {
"asn": "65032",
"holdtime": "180",
"keepalive": "60",
"local_addr": "10.0.1.13",
"name": "leaf1",
"nhopself": "0",
"rrclient": "0"
},
"172.16.0.10": {
"asn": "65033",
"holdtime": "180",
"keepalive": "60",
"local_addr": "10.0.1.13",
"name": "leaf2",
"nhopself": "0",
"rrclient": "0"
Under "DEVICE_METADATA" change "bgp_asn": "65100" to spine1 asn number
"DEVICE_METADATA": {
"localhost": {
"bgp_asn": "65030",
Close the file and run sudo config reload -y
Repeat the same steps for spine2, leaf1 and leaf2 by replacing the "IP addresses" and "asn"
verification
show ip bgp summary
spine1
Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 172.16.0.2 4 65032 1566 1569 0 0 0 1d01h57m 3 leaf1 172.16.0.10 4 65033 1534 1543 0 0 0 1d01h27m 2 leaf2 Total number of neighbors 2
spine2
Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 172.16.0.6 4 65032 1548 1548 0 0 0 1d01h40m 2 leaf1 172.16.0.14 4 65033 1543 1544 0 0 0 1d01h28m 3 leaf2 Total number of neighbors 2
leaf1
Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 172.16.0.1 4 65030 1562 1561 0 0 0 1d01h53m 2 Spine1 172.16.0.5 4 65031 1543 1647 0 0 0 1d01h35m 2 spine2 Total number of neighbors 2
leaf2
Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 172.16.0.9 4 65030 1539 1537 0 0 0 1d01h30m 3 spine1 172.16.0.13 4 65031 1538 1537 0 0 0 1d01h30m 3 spine2
Known limitations
- It is not possible to configure a DNS resolver, all configuration needs to be done using IPs
- It is not possible to change the syslog facility
cgexec -g l3mdev:mgmtis supposed to run commands from the mgmt VRF, but fails with "cgroup change of group failed"- "ip vrf exec <name> <command>" does seem to work for this. (name being the vrf name, "mgmt" in this case.)