You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Wikimedia Cloud Services team/EnhancementProposals/2020 Network refresh/Implementation details: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Arturo Borrero Gonzalez
(create page)
 
imported>Arturo Borrero Gonzalez
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This page contains the '''implementation details''' for the [[Wikimedia_Cloud_Services_team/EnhancementProposals/2020_Network_refresh | 2020 Network refresh project]].
This page contains the '''implementation details''' for the [[Wikimedia_Cloud_Services_team/EnhancementProposals/2020_Network_refresh | 2020 Network refresh project]].


==== eqiad ====
== eqiad ==


[[File:Cloudgw-stage_3.png|900px]]
[[File:Cloudgw-stage_3.png|900px]]
Line 7: Line 7:
In the eqiad datacenter, related to the eqiad1 openstack deployment.
In the eqiad datacenter, related to the eqiad1 openstack deployment.


===== specs for eqiad1 =====
=== specs for eqiad1 ===


On '''cloudgw''' side, each server:
On '''cloudgw''' side, each server:
Line 25: Line 25:
* Juniper QFX5100 switches with L3 routing licenses
* Juniper QFX5100 switches with L3 routing licenses


===== network setup in eqiad1 =====
=== network setup in eqiad1 ===


====== allocations ======
==== allocations ====


IPv4 allocations:
IPv4 allocations:
Line 58: Line 58:
</syntaxhighlight>
</syntaxhighlight>


====== <code>stage 0</code> starting network setup ======
==== <code>stage 0</code> starting point, current network setup ====
 
'''TODO:''' for reference, include here some bits about the starting setup of the network?


{| class="wikitable"
{| class="wikitable"
Line 97: Line 95:
|}
|}


====== <code>stage 1</code>: Route cloud-hosts vlan through cloudsw ======


The cloud-hosts vlan, which is part of the production realm, is curently routed on cr1/2-eqiad:ae2.1118. Which are the interfaces facing asw2-b-eqiad.
The cloud-hosts vlan, which is part of the production realm, is curently routed on cr1/2-eqiad:ae2.1118. Which are the interfaces facing asw2-b-eqiad.
Line 103: Line 100:
In the optic of better separation of WMCS and production realm, that routing should be moved to cr1/2-eqiad:xe-3/0/4.1118, the interfaces facing cloudsw.
In the optic of better separation of WMCS and production realm, that routing should be moved to cr1/2-eqiad:xe-3/0/4.1118, the interfaces facing cloudsw.


This will contribute to goals '''(A)''' and '''(C)''' of the '''cloudsw''' project.
This already contributes to goals '''(A)''' and '''(C)'''. This was a low complexity change. See https://phabricator.wikimedia.org/T261866 for the implementation.


This is a low complexity change. See https://phabricator.wikimedia.org/T261866 for the implementation.
==== <code>stage 1</code>: validate cloudgw changes in codfw ====


====== <code>stage 2A</code>: enable L3 routing on '''cloudsw''' nodes ======
This is a NOOP in the eqiad DC.
This will contribute to goals '''(A), (B)''', '''(C)''' and '''(D)''' of the '''cloudsw''' project.
 
==== <code>stage 2</code>: enable L3 routing on '''cloudsw''' nodes ====
This will contribute to goals '''(A), (B)''', '''(C)''' and '''(D)''' of the project.
[[File:WMCS network-L2 L3.png|thumb]]
[[File:WMCS network-L2 L3.png|thumb]]


Steps and implementation on https://phabricator.wikimedia.org/T265288


Steps (to be moved to a task for implementation):
# Baseline configuration
## Cloudsw vlans (L2) - 1102, 1103, 1104, 1120
## iBGP and OSPF between cloudsw
## eBGP between core routers and cloudsw (advertise 208.80.155.88/29, 185.15.56.0/24 and 172.16.0.0/21, receive 0/0)
## Static route for 185.15.56.0/25 and 172.16.0.0/21 on cloudsw
## Firewall filters - lo, cloud-in4 (on core routers)
## Test connectivity
# cloud-instances-transport migration (downtime required [!])
## Ensure cr1 is VRRP master for all vlans, including 1120
## Move cr2:ae2.1120 to cloudsw1-d5:irb.1120
## Test cr1:ae2.1120 to cloudsw1-d5:irb.1120 connectivity (and VRRP sync)
## [!] Move vlan 1120 VRRP master to cloudsw1-d5:irb.1120
## [!] Remove static routes for 185.15.56.0/25 and 172.16.0.0/21 on core routers
## Test connectivity
## Move cr1:ae2.1120 to cloudsw1-c8:irb.1120
## Cleanup (remove passive OSPF, trunked vlans, update Netbox)
# Renumber cloud-instances-transport (downtime required [!]) [Could be done when introducing cloudgw] similar to https://phabricator.wikimedia.org/T207663
## Configure 85.15.56.240/29 IPs on all devices
## [!] Reconfigure cloudnet with new gateway IP (to be confirmed)
## Update static routes on cloudsw to point to new VIP
## Cleanup 208.80.155.88/29 IPs and advertisement (+Netbox)
At this stage:
At this stage:
{| class="wikitable"
{| class="wikitable"
Line 185: Line 162:
<nowiki>*</nowiki> To be removed when hosts are moved away from that device
<nowiki>*</nowiki> To be removed when hosts are moved away from that device


====== <code>stage 2B</code>: enable L3 routing on '''cloudgw''' nodes ======
==== <code>stage 3</code>: enable L3 routing on '''cloudgw''' nodes ====
TBD


====== <code>stage 3</code> final status for all main network components ======
TBD
*[[File:Cloudgw_L2_stage_3_eqiad.png|right|400px]]connectivity between '''cloudgw''' and the '''cloud-hosts1-b-eqiad''' subnet.
*[[File:Cloudgw_L2_stage_3_eqiad.png|right|400px]]connectivity between '''cloudgw''' and the '''cloud-hosts1-b-eqiad''' subnet.
** L3:
** L3:
Line 229: Line 202:
** L3: allocate two new interco /31s prefixes (208.80.154.210/31 and 208.80.154.212/31), configure eBGP in <code>stage 2A</code>
** L3: allocate two new interco /31s prefixes (208.80.154.210/31 and 208.80.154.212/31), configure eBGP in <code>stage 2A</code>


==== codfw ====
== codfw ==
 
{{tracked|T261724}}
{{tracked|T263622}}


[[File:Cloudgw-L3 stage 3 codfw(1).png | 900px ]]
[[File:Cloudgw-L3 stage 3 codfw(1).png | 900px ]]
{{tracked|T261724}}
 


In the codfw datacenter, related to the codfw1dev openstack deployment.
In the codfw datacenter, related to the codfw1dev openstack deployment.


===== specs for codfw1dev =====
=== specs for codfw1dev ===


For '''cloudgw''',  repurpose [https://netbox.wikimedia.org/dcim/devices/1774/ labtestvirt2003] as '''cloudgw2001-dev'''.
For '''cloudgw''',  repurpose [https://netbox.wikimedia.org/dcim/devices/1774/ labtestvirt2003] as '''cloudgw2001-dev'''.
Line 242: Line 218:
For '''cloudsw''', we assume we wont have the device anytime soon.
For '''cloudsw''', we assume we wont have the device anytime soon.


===== network setup in codfw1dev =====
=== network setup in codfw1dev ===


Specific configuration details for each stage.
Specific configuration details for each stage.


====== allocations ======
==== allocations ====


IPv4 allocations:
IPv4 allocations:
Line 253: Line 229:
185.15.57.0/24
185.15.57.0/24
     185.15.57.0/29 - Openstack instances NAT (floating IPs)
     185.15.57.0/29 - Openstack instances NAT (floating IPs)
     185.15.57.8/29 - reserved for the above growth
     185.15.57.8/30 - 2107 - cloud-gw-transport-codfw (cloudgw <-> neutron)
185.15.57.16/28 - unused
185.15.57.16/28 - unused
185.15.57.32/27 - unused
185.15.57.32/27 - unused
185.15.57.64/26 - unused
185.15.57.64/26 - unused
     185.15.57.128/25 - infrastructure
     185.15.57.128/25 - infrastructure
        185.15.57.128/29 - 2120 - cloud-instances-transport1-b-codfw (cr-codfw <-> cloudgw)
208.80.153.184/29 - 2120 - cloud-instances-transport1-b-codfw (cr-codfw <-> cloudgw)
        185.15.57.144/29 - 2107 - cloud-gw-transport-codfw (cloudgw <-> neutron)
</syntaxhighlight>
</syntaxhighlight>


Line 266: Line 241:
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
2105 - cloud-instances1-codfw (172.16.128.0/24)
2105 - cloud-instances1-codfw (172.16.128.0/24)
2107 - cloud-gw-transport-codfw (cloudgw <-> neutron) (185.15.57.144/29)
2107 - cloud-gw-transport-codfw (cloudgw <-> neutron) (185.15.57.8/31)
2118 - cloud-hosts1-codfw (10.192.20.0/24)
2118 - cloud-hosts1-codfw (10.192.20.0/24)
2120 - cloud-instances-transport1-codfw (cr-codfw <-> cloudgw) (185.15.57.128/29)
2120 - cloud-instances-transport1-codfw (cr-codfw <-> cloudgw) (208.80.153.184/29)
</syntaxhighlight>
</syntaxhighlight>


 
==== <code>stage 0</code>: starting point, current network setup ====
====== <code>stage 0</code> starting network setup ======


'''TODO:''' for reference, include here some bits about the starting setup of the network?
'''TODO:''' for reference, include here some bits about the starting setup of the network?


====== <code>stage 1</code>: Route cloud-hosts vlan through cloudsw ======
==== <code>stage 1</code>: validate cloudgw changes in codfw ====


We don't have hardware for cloudsw in codfw. This stage is NOOP.
Given we don't have hardware for testing the cloudsw setup in codfw, we assume we are working with core routers and asw.


====== <code>stage 2B</code>: enable L3 routing on '''cloudsw''' nodes ======
In this stage, we validate all the cloudgw changes that will be later implemented in eqiad. We use the '''labtestvirt2003.codfw.wmnet''' server acting as '''cloudgw''' in this PoC.
 
We don't have hardware for cloudsw in codfw. This stage is NOOP.
 
====== <code>stage 2A</code>: enable L3 routing on '''cloudgw''' nodes ======
 
'''TODO:''' describe here the PoC we will be doing with '''labtestvirt2003'''.
 
====== <code>stage 3</code> final status for all main network components ======
 
'''TODO:'''due to lack of resources in codfw we don't have yet and estimation of when this stage can be implemented.


* connectivity between '''cloudgw''' and the '''cloud-hosts1-b-codfw''' subnet.
* connectivity between '''cloudgw''' and the '''cloud-hosts1-b-codfw''' subnet.
** L3:
** L3:
*** a single IP address allocated by standard methods for ssh management, puppet, monitoring, etc. Gateway for this subnet lives in '''cloudsw'''.
*** a single IP address allocated by standard methods for ssh management, puppet, monitoring, etc. Gateway for this subnet lives in the core router (we don't have cloudsw)
** L2:
** L2:
*** '''cloudgw''' has 2 NICs bonded/teamed/aggregated and then trunked with 3 vlans:
*** '''cloudgw''' has 2 NICs, the control plane one (eno1) connected to:
**** [https://netbox.wikimedia.org/ipam/prefixes/147/ cloud-hosts1-b-codfw] ([https://netbox.wikimedia.org/ipam/vlans/113/ vlan 2118]) '''10.192.20.0/24'''
**** [https://netbox.wikimedia.org/ipam/prefixes/147/ cloud-hosts1-b-codfw] ([https://netbox.wikimedia.org/ipam/vlans/113/ vlan 2118]) '''10.192.20.0/24''' (untagged)
**** [https://netbox.wikimedia.org/ipam/prefixes/39/ cloud-instances-transport1-b-codfw] (cloudsw<->cloudgw) ([https://netbox.wikimedia.org/ipam/vlans/114/ vlan 2120]) '''208.80.153.184/29'''
**** cloud-gw-transport-codfw (cloudgw <-> neutron) (vlan 2107) '''185.15.57.144/29'''


* connectivity between Neutron (cloudnet) and '''cloudgw''':
* connectivity between Neutron (cloudnet) and '''cloudgw''':
** L3:
** L3:
*** '''cloudnet''' keeps the current connection to the '''cloud-hosts1-b-codfw''' subnet for ssh management, puppet, monitoring, etc. Gateway for this subnet lives in '''cloudsw'''.
*** '''cloudnet''' keeps the current connection to the '''cloud-hosts1-b-codfw''' subnet for ssh management, puppet, monitoring, etc. Gateway for this subnet lives in the core router (we don't have cloudsw).
*** drop the current [https://netbox.wikimedia.org/ipam/prefixes/39/ cloud-instances-transport1-b-codfw] ([https://netbox.wikimedia.org/ipam/vlans/114/ vlan 2120]) '''208.80.153.184/29'''
*** drop (or leave unused) the current [https://netbox.wikimedia.org/ipam/prefixes/39/ cloud-instances-transport1-b-codfw] ([https://netbox.wikimedia.org/ipam/vlans/114/ vlan 2120]) '''208.80.153.184/29'''
*** add cloud-gw-transport-codfw (cloudgw <-> neutron) (vlan 2107) '''185.15.57.144/29'''
*** add [https://netbox.wikimedia.org/ipam/prefixes/353/ cloud-gw-transport-codfw] (cloudgw <-> neutron) ([https://netbox.wikimedia.org/ipam/vlans/144/ vlan 2107]) '''185.15.57.8/30'''
*** keep the current cloud-instances2-b-codfw (vlan 2105) '''172.16.128.0/24'''
*** keep the current cloud-instances2-b-codfw (vlan 2105) '''172.16.128.0/24'''
** L2:
** L2:
*** '''cloudnet''' keep 2 NICs, each with different setup:
*** '''cloudnet''' keep 2 NICs, each with different setup:
**** [https://netbox.wikimedia.org/ipam/prefixes/147/ cloud-hosts1-b-codfw] ([https://netbox.wikimedia.org/ipam/vlans/113/ vlan 2118]) '''10.192.20.0/24'''
**** [https://netbox.wikimedia.org/ipam/prefixes/147/ cloud-hosts1-b-codfw] ([https://netbox.wikimedia.org/ipam/vlans/113/ vlan 2118]) '''10.192.20.0/24'''
**** other trunked with vlan 2105 and vlan 2107 (cloud-virt-instance-trunk).
**** other trunked with at least vlan 2105 and vlan 2107 (cloud-virt-instance-trunk).
*** '''cloudgw''' has 2 NICs bonded/teamed/aggregated and then trunked with 3 vlans:
*** '''cloudgw''' has 2 NICs, the data plane one (eno2) being trunked with:
**** [https://netbox.wikimedia.org/ipam/prefixes/147/ cloud-hosts1-b-codfw] ([https://netbox.wikimedia.org/ipam/vlans/113/ vlan 2118]) '''10.192.20.0/24'''
**** [https://netbox.wikimedia.org/ipam/prefixes/39/ cloud-instances-transport1-b-codfw] (cr-codfw<->cloudgw) ([https://netbox.wikimedia.org/ipam/vlans/114/ vlan 2120]) '''208.80.153.184/29'''
**** [https://netbox.wikimedia.org/ipam/prefixes/39/ cloud-instances-transport1-b-codfw] (cloudsw<->cloudgw) ([https://netbox.wikimedia.org/ipam/vlans/114/ vlan 2120]) '''208.80.153.184/29'''
**** [https://netbox.wikimedia.org/ipam/prefixes/353/ cloud-gw-transport-codfw] (cloudgw <-> neutron) ([https://netbox.wikimedia.org/ipam/vlans/144/ vlan 2107]) '''185.15.57.8/30'''
**** cloud-gw-transport-codfw (cloudgw <-> neutron) (vlan 2107) '''185.15.57.144/29'''


* connectivity between '''cloudgw''' and '''cr-codfw''':
* connectivity between '''cloudgw''' and '''cr-codfw''':
** L3:
** L3:
*** relocate cloud-instances-transport1-codfw (cr-codfw <-> cloudgw) (185.15.57.128/29) vlan 2120
*** interconnect using [https://netbox.wikimedia.org/ipam/prefixes/39/ cloud-instances-transport1-b-codfw] (cr-codfw<->cloudgw) ([https://netbox.wikimedia.org/ipam/vlans/114/ vlan 2120]) '''208.80.153.184/29'''
** L2:
** L2:
*** '''cloudgw''' has 2 NICs bonded/teamed/aggregated and then trunked with 3 vlans:
*** '''cloudgw''' has 2 NICs, the data plane one (eno2) being trunked with:
**** [https://netbox.wikimedia.org/ipam/prefixes/147/ cloud-hosts1-b-codfw] ([https://netbox.wikimedia.org/ipam/vlans/113/ vlan 2118]) '''10.192.20.0/24'''
**** [https://netbox.wikimedia.org/ipam/prefixes/39/ cloud-instances-transport1-b-codfw] (cr-codfw<->cloudgw) ([https://netbox.wikimedia.org/ipam/vlans/114/ vlan 2120]) '''208.80.153.184/29'''
**** [https://netbox.wikimedia.org/ipam/prefixes/39/ cloud-instances-transport1-b-codfw] (cloudsw<->cloudgw) ([https://netbox.wikimedia.org/ipam/vlans/114/ vlan 2120]) '''208.80.153.184/29'''
**** [https://netbox.wikimedia.org/ipam/prefixes/353/ cloud-gw-transport-codfw] (cloudgw <-> neutron) ([https://netbox.wikimedia.org/ipam/vlans/144/ vlan 2107]) '''185.15.57.8/30'''
**** cloud-gw-transport-codfw (cloudgw <-> neutron) (vlan 2107) '''185.15.57.144/29'''
 
''' neutron operations '''
 
* define new subnet object
* update external fixed IP address, now using an address from vlan 2107 cloud-gw-transport-codfw (185.15.57.8/30)
* disable SNAT (now done in cloudgw)
 
<syntaxhighlight lang="shell-session">
root@cloudcontrol2001-dev:~# openstack router show cloudinstances2b-gw -f yaml
admin_state_up: UP
availability_zone_hints: ''
availability_zones: nova
created_at: '2018-03-29T14:18:50Z'
description: ''
distributed: false
external_gateway_info: '{"network_id": "57017d7c-3817-429a-8aa3-b028de82cdcc", "enable_snat":
  true, "external_fixed_ips": [{"subnet_id": "31214392-9ca5-4256-bff5-1e19a35661de",
  "ip_address": "208.80.153.190"}]}'
flavor_id: null
ha: true
id: 5712e22e-134a-40d3-a75a-1c9b441717ad
interfaces_info: '[{"port_id": "21e10025-d464-45a6-82ac-25894e9164e4", "ip_address":
  "172.16.128.1", "subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668"}, {"port_id":
  "5dc9c3b7-245f-43f7-8db1-baf7bdf175fd", "ip_address": "169.254.192.4", "subnet_id":
  "651250de-53ca-4487-97ce-e6f65dc4b8ec"}, {"port_id": "727a378d-3558-4132-933a-e2e72c28e532",
  "ip_address": "169.254.192.5", "subnet_id": "651250de-53ca-4487-97ce-e6f65dc4b8ec"}]'
name: cloudinstances2b-gw
project_id: admin
revision_number: 2
routes: ''
status: ACTIVE
tags: ''
updated_at: '2019-10-02T10:30:11Z'
root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.9 --no-dhcp --subnet-range 185.15.57.8/30 cloud-gw-transport-codfw
+-------------------+--------------------------------------+
| Field            | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 185.15.57.10-185.15.57.10            |
| cidr              | 185.15.57.8/30                      |
| created_at        | 2020-10-09T08:48:11Z                |
| description      |                                      |
| dns_nameservers  |                                      |
| enable_dhcp      | False                                |
| gateway_ip        | 185.15.57.9                          |
| host_routes      |                                      |
| id                | 2596edb4-5a40-41b9-9e67-f1f9e40e329c |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                |
| ipv6_ra_mode      | None                                |
| name              | cloud-gw-transport-codfw            |
| network_id        | 57017d7c-3817-429a-8aa3-b028de82cdcc |
| project_id        | admin                                |
| revision_number  | 0                                    |
| segment_id        | None                                |
| service_types    |                                      |
| subnetpool_id    | None                                |
| tags              |                                      |
| updated_at        | 2020-10-09T08:48:11Z                |
+-------------------+--------------------------------------+
root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-gw-transport-codfw,ip-address=185.15.57.10 cloudinstances2b-gw
root@cloudcontrol2001-dev:~# openstack subnet delete cloud-instances-transport1-b-codfw
root@cloudcontrol2001-dev:~# openstack router set --disable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw
</syntaxhighlight>
 
That command disables both '''routing_source_ip''' and '''dmz_cidr''' according to this diff (note the specific rules are missing):
 
{{Collapse top|iptables-save diff}}
<syntaxhighlight lang="diff">
--- enabled.txt 2020-09-23 10:20:45.373366952 +0000
+++ disabled.txt 2020-09-23 10:20:56.397471627 +0000
@@ -1,4 +1,4 @@
-# Generated by iptables-save v1.8.5 on Wed Sep 23 10:20:45 2020
+# Generated by iptables-save v1.8.5 on Wed Sep 23 10:20:56 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
@@ -19,9 +19,10 @@
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-scope -o qr-21e10025-d4 -m mark ! --mark 0x4010000/0xffff0000 -j DROP
+-A neutron-l3-agent-scope -o qg-1290224c-b1 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
COMMIT
-# Completed on Wed Sep 23 10:20:45 2020
-# Generated by iptables-save v1.8.5 on Wed Sep 23 10:20:45 2020
+# Completed on Wed Sep 23 10:20:56 2020
+# Generated by iptables-save v1.8.5 on Wed Sep 23 10:20:56 2020
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
@@ -45,7 +46,6 @@
-A neutron-l3-agent-FORWARD -s 172.16.128.19/32 -j neutron-l3-agent-float-snat
-A neutron-l3-agent-FORWARD -s 172.16.128.20/32 -j neutron-l3-agent-float-snat
-A neutron-l3-agent-FORWARD -s 172.16.128.26/32 -j neutron-l3-agent-float-snat
--A neutron-l3-agent-POSTROUTING -o qg-1290224c-b1 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-mark
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-scope
-A neutron-l3-agent-PREROUTING -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
@@ -56,12 +56,11 @@
-A neutron-l3-agent-floatingip -d 185.15.57.2/32 -j MARK --set-xmark 0x4010000/0xffff0000
-A neutron-l3-agent-floatingip -d 185.15.57.4/32 -j MARK --set-xmark 0x4010000/0xffff0000
-A neutron-l3-agent-floatingip -d 185.15.57.6/32 -j MARK --set-xmark 0x4010000/0xffff0000
--A neutron-l3-agent-mark -i qg-1290224c-b1 -j MARK --set-xmark 0x2/0xffff
-A neutron-l3-agent-scope -i qr-21e10025-d4 -j MARK --set-xmark 0x4010000/0xffff0000
-A neutron-l3-agent-scope -i qg-1290224c-b1 -j MARK --set-xmark 0x4000000/0xffff0000
COMMIT
-# Completed on Wed Sep 23 10:20:45 2020
-# Generated by iptables-save v1.8.5 on Wed Sep 23 10:20:45 2020
+# Completed on Wed Sep 23 10:20:56 2020
+# Generated by iptables-save v1.8.5 on Wed Sep 23 10:20:56 2020
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
@@ -80,10 +79,6 @@
-A neutron-l3-agent-OUTPUT -d 185.15.57.2/32 -j DNAT --to-destination 172.16.128.19
-A neutron-l3-agent-OUTPUT -d 185.15.57.4/32 -j DNAT --to-destination 172.16.128.20
-A neutron-l3-agent-OUTPUT -d 185.15.57.6/32 -j DNAT --to-destination 172.16.128.26
--A neutron-l3-agent-POSTROUTING -s 208.80.153.190/32 -j ACCEPT
--A neutron-l3-agent-POSTROUTING -s 172.16.128.0/24 -d 10.0.0.0/8 -j ACCEPT
--A neutron-l3-agent-POSTROUTING -s 172.16.128.0/24 -d 208.80.152.0/22 -j ACCEPT
--A neutron-l3-agent-POSTROUTING ! -i qg-1290224c-b1 ! -o qg-1290224c-b1 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 185.15.57.2/32 -j DNAT --to-destination 172.16.128.19
-A neutron-l3-agent-PREROUTING -d 185.15.57.4/32 -j DNAT --to-destination 172.16.128.20
@@ -92,12 +87,10 @@
-A neutron-l3-agent-float-snat -s 172.16.128.20/32 -j SNAT --to-source 185.15.57.4 --random-fully
-A neutron-l3-agent-float-snat -s 172.16.128.26/32 -j SNAT --to-source 185.15.57.6 --random-fully
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
--A neutron-l3-agent-snat -o qg-1290224c-b1 -j SNAT --to-source 185.15.57.1 --random-fully
--A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 208.80.153.190 --random-fully
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
-# Completed on Wed Sep 23 10:20:45 2020
-# Generated by iptables-save v1.8.5 on Wed Sep 23 10:20:45 2020
+# Completed on Wed Sep 23 10:20:56 2020
+# Generated by iptables-save v1.8.5 on Wed Sep 23 10:20:56 2020
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
@@ -106,4 +99,4 @@
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
COMMIT
-# Completed on Wed Sep 23 10:20:45 2020
+# Completed on Wed Sep 23 10:20:56 2020
</syntaxhighlight>
{{Collapse bottom}}
 
===== <code>stage 1</code>: validation checklist =====
 
This list should help validate the new model works as expected.
 
# The cloudgw deployment is reproducible, server can be properly reimaged
# Traffic is isolated between data plane and control plane networks
# VM (no floating IP) contacting the internet gets NATed using routing_source_ip
# VM (no floating IP) contacting an address covered by dmz_cidr doesn't get NATed using routing_source_ip
# VM (using floating IP) isn't affected by either routing_source_ip or dmz_cidr
# VM (no floating IP) can contact auth DNS server
# VM (no floating IP) can contact rec DNS server
# VM (using floating IP) can contact auth DNS server
# VM (using floating IP) can contact rec DNS server
# VM (no floating IP) can contact LDAP server
# VM (using floating IP) can contact LDAP server
# VM (no floating IP) can mount NFS (dumps)
# VM (floating IP) can mount NFS (dumps)
# VM (no floating IP) can mount NFS (scratch)
# VM (floating IP) can mount NFS (scratch)
# VM (no floating IP) can mount NFS (maps)
# VM (floating IP) can mount NFS (maps)
# VM (no floating IP) can mount NFS (tools)
# VM (floating IP) can mount NFS (tools)
# VM (no floating IP) can connect to wiki-replicas
# VM (floating IP) can connect to wiki-replicas
# VM (no floating IP) can use openstack endpoints
# VM (floating IP) can use openstack endpoints
# All VMs can use the internal openstack metadata address
# puppetmasters VM work as expected
# security groups work as expected
 
==== <code>stage 2</code>: enable L3 routing on '''cloudsw''' nodes ====
 
Given we don't have hardware for testing the cloudsw setup in codfw, we assume we are working with core routers and asw.
 
Therefore this stage is a NOOP in the codfw datacenter.
 
==== <code>stage 3</code>: enable L3 routing on '''cloudgw''' nodes ====
 
All the codfw changes were done in the <code>stage 1</code>, therefore this stage is a NOOP in the codfw datacenter.

Latest revision as of 11:26, 22 October 2020

This page contains the implementation details for the 2020 Network refresh project.

eqiad

Cloudgw-stage 3.png

In the eqiad datacenter, related to the eqiad1 openstack deployment.

specs for eqiad1

On cloudgw side, each server:

  • Hardware, misc box
    • CPU: 16 CPU
    • RAM: 32 GB
    • Disk: 500GB
    • 2 x 10Gbps NICs. NICs are bonded/teamed/aggregated for redundancy.
  • Software
    • standard puppet management
    • prometheus metrics, icinga monitoring
    • netfilter for NAT/firewalling
    • keepalived or corosync/pacemaker for HA

On cloudsw side, each device:

  • Juniper QFX5100 switches with L3 routing licenses

network setup in eqiad1

allocations

IPv4 allocations:

185.15.56.0/24
    185.15.56.0/25 - Openstack instances NAT
    185.15.56.128/26 - reserved for the above groth
    185.15.56.192/27 - unused
    185.15.56.224/28 - unused
    185.15.56.240/28 - infrastructure
        185.15.56.240/29 - 1120 - cloud-instances-transport1
        185.15.56.248/31 - 1104 - cloudsw1-c8<->cloudsw1-d5 - cloud-xlink1
        185.15.56.250/31 - unused
        185.15.56.252/30 - loopbacks

VLAN allocations:

1102 - cr1<->cloudsw1-c8 - cloud-transit1-eqiad
1103 - cr2<->cloudsw1-d5 - cloud-transit2-eqiad
1104 - cloudsw1-c8<->cloudsw1-d5 - cloud-xlink1-eqiad
1105 - cloud-instances1-eqiad
1106 - cloud-storage1-eqiad
1107 - cloudsw1<->cloudgw - cloud-gw-transport-eqiad ?
1118 - cloud-hosts1-eqiad

1120 - cloud-instances-transport1-eqiad

stage 0 starting point, current network setup

VLAN Switched on L2 Members L3 Gateway (“to internet”)
cloud-hosts1-eqiad asw2-b cr1/2

all cloudvirt eth0

all Ceph OSD eth0

cr1/2 (via asw2-b)
cloud-instances2-eqiad asw2-b all cloud VPS

all cloudvirt eth1

cloudnet1003/1004 eth1

cloudnet1003/1004 eth1
cloud-instances-transport1-eqiad asw2-b cloudnet1003/1004 eth0 cr1/2
cloud-storage1-eqiad asw2-b all cloudcephosd eth1 (none)


The cloud-hosts vlan, which is part of the production realm, is curently routed on cr1/2-eqiad:ae2.1118. Which are the interfaces facing asw2-b-eqiad.

In the optic of better separation of WMCS and production realm, that routing should be moved to cr1/2-eqiad:xe-3/0/4.1118, the interfaces facing cloudsw.

This already contributes to goals (A) and (C). This was a low complexity change. See https://phabricator.wikimedia.org/T261866 for the implementation.

stage 1: validate cloudgw changes in codfw

This is a NOOP in the eqiad DC.

stage 2: enable L3 routing on cloudsw nodes

This will contribute to goals (A), (B), (C) and (D) of the project.

WMCS network-L2 L3.png

Steps and implementation on https://phabricator.wikimedia.org/T265288

At this stage:

VLAN Switched on L2 Members L3 Gateway (“to internet”)
cloud-hosts1-eqiad asw2-b*

cloudsw

cr1/2

all cloudvirt eth0

all Ceph OSD eth0

cr1/2 (via cloudsw)
cloud-instances2-eqiad asw2-b*

cloudsw

all cloud VPS

all cloudvirt eth1

cloudnet1003/1004 eth1

cloudnet1003/1004 eth1
cloud-instances-transport1-eqiad asw2-b*

cloudsw

cloudsw

cloudnet1003/1004 eth0

cloudsw
cloud-transit1/2-eqiad cloudsw cr1/2

cloudsw

cr1/2
cloud-storage1-eqiad asw2-b*

cloudsw

all cloudcephosd eth1 (none)

* To be removed when hosts are moved away from that device

stage 3: enable L3 routing on cloudgw nodes

  • Cloudgw L2 stage 3 eqiad.png
    connectivity between cloudgw and the cloud-hosts1-b-eqiad subnet.
    • L3:
      • a single IP address allocated by standard methods for ssh management, puppet, monitoring, etc. Gateway for this subnet lives on core routers, but is switches through cloudgw after stage 1.
    • L2:
  • connectivity between cloudgw and cloudsw:
    • L3:
      • allocate new transport range and vlan 11XX.
      • static routes between cloudgw and cloudsw
    • L2:
  • connectivity between cloudsw and prod core router:
    • L1: cloudsw are directly connected to the prod core routers using 1x10G port each
    • L2: 2 vlans are trunked between the two sides: vlan 1118 (cloud-hosts) and 1102 (public interco vlan)
    • L3: allocate two new interco /31s prefixes (208.80.154.210/31 and 208.80.154.212/31), configure eBGP in stage 2A

codfw

Cloudgw-L3 stage 3 codfw(1).png


In the codfw datacenter, related to the codfw1dev openstack deployment.

specs for codfw1dev

For cloudgw, repurpose labtestvirt2003 as cloudgw2001-dev.

For cloudsw, we assume we wont have the device anytime soon.

network setup in codfw1dev

Specific configuration details for each stage.

allocations

IPv4 allocations:

185.15.57.0/24
    185.15.57.0/29 - Openstack instances NAT (floating IPs)
    185.15.57.8/30 - 2107 - cloud-gw-transport-codfw (cloudgw <-> neutron)
	185.15.57.16/28 - unused
	185.15.57.32/27 - unused
	185.15.57.64/26 - unused
    185.15.57.128/25 - infrastructure
208.80.153.184/29 - 2120 - cloud-instances-transport1-b-codfw (cr-codfw <-> cloudgw)

VLAN allocations:

2105 - cloud-instances1-codfw (172.16.128.0/24)
2107 - cloud-gw-transport-codfw (cloudgw <-> neutron) (185.15.57.8/31)
2118 - cloud-hosts1-codfw (10.192.20.0/24)
2120 - cloud-instances-transport1-codfw (cr-codfw <-> cloudgw) (208.80.153.184/29)

stage 0: starting point, current network setup

TODO: for reference, include here some bits about the starting setup of the network?

stage 1: validate cloudgw changes in codfw

Given we don't have hardware for testing the cloudsw setup in codfw, we assume we are working with core routers and asw.

In this stage, we validate all the cloudgw changes that will be later implemented in eqiad. We use the labtestvirt2003.codfw.wmnet server acting as cloudgw in this PoC.

  • connectivity between cloudgw and the cloud-hosts1-b-codfw subnet.
    • L3:
      • a single IP address allocated by standard methods for ssh management, puppet, monitoring, etc. Gateway for this subnet lives in the core router (we don't have cloudsw)
    • L2:

neutron operations

  • define new subnet object
  • update external fixed IP address, now using an address from vlan 2107 cloud-gw-transport-codfw (185.15.57.8/30)
  • disable SNAT (now done in cloudgw)
root@cloudcontrol2001-dev:~# openstack router show cloudinstances2b-gw -f yaml
admin_state_up: UP
availability_zone_hints: ''
availability_zones: nova
created_at: '2018-03-29T14:18:50Z'
description: ''
distributed: false
external_gateway_info: '{"network_id": "57017d7c-3817-429a-8aa3-b028de82cdcc", "enable_snat":
  true, "external_fixed_ips": [{"subnet_id": "31214392-9ca5-4256-bff5-1e19a35661de",
  "ip_address": "208.80.153.190"}]}'
flavor_id: null
ha: true
id: 5712e22e-134a-40d3-a75a-1c9b441717ad
interfaces_info: '[{"port_id": "21e10025-d464-45a6-82ac-25894e9164e4", "ip_address":
  "172.16.128.1", "subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668"}, {"port_id":
  "5dc9c3b7-245f-43f7-8db1-baf7bdf175fd", "ip_address": "169.254.192.4", "subnet_id":
  "651250de-53ca-4487-97ce-e6f65dc4b8ec"}, {"port_id": "727a378d-3558-4132-933a-e2e72c28e532",
  "ip_address": "169.254.192.5", "subnet_id": "651250de-53ca-4487-97ce-e6f65dc4b8ec"}]'
name: cloudinstances2b-gw
project_id: admin
revision_number: 2
routes: ''
status: ACTIVE
tags: ''
updated_at: '2019-10-02T10:30:11Z'
root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.9 --no-dhcp --subnet-range 185.15.57.8/30 cloud-gw-transport-codfw
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 185.15.57.10-185.15.57.10            |
| cidr              | 185.15.57.8/30                       |
| created_at        | 2020-10-09T08:48:11Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | False                                |
| gateway_ip        | 185.15.57.9                          |
| host_routes       |                                      |
| id                | 2596edb4-5a40-41b9-9e67-f1f9e40e329c |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | cloud-gw-transport-codfw             |
| network_id        | 57017d7c-3817-429a-8aa3-b028de82cdcc |
| project_id        | admin                                |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2020-10-09T08:48:11Z                 |
+-------------------+--------------------------------------+
root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-gw-transport-codfw,ip-address=185.15.57.10 cloudinstances2b-gw
root@cloudcontrol2001-dev:~# openstack subnet delete cloud-instances-transport1-b-codfw
root@cloudcontrol2001-dev:~# openstack router set --disable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw

That command disables both routing_source_ip and dmz_cidr according to this diff (note the specific rules are missing):

stage 1: validation checklist

This list should help validate the new model works as expected.

  1. The cloudgw deployment is reproducible, server can be properly reimaged
  2. Traffic is isolated between data plane and control plane networks
  3. VM (no floating IP) contacting the internet gets NATed using routing_source_ip
  4. VM (no floating IP) contacting an address covered by dmz_cidr doesn't get NATed using routing_source_ip
  5. VM (using floating IP) isn't affected by either routing_source_ip or dmz_cidr
  6. VM (no floating IP) can contact auth DNS server
  7. VM (no floating IP) can contact rec DNS server
  8. VM (using floating IP) can contact auth DNS server
  9. VM (using floating IP) can contact rec DNS server
  10. VM (no floating IP) can contact LDAP server
  11. VM (using floating IP) can contact LDAP server
  12. VM (no floating IP) can mount NFS (dumps)
  13. VM (floating IP) can mount NFS (dumps)
  14. VM (no floating IP) can mount NFS (scratch)
  15. VM (floating IP) can mount NFS (scratch)
  16. VM (no floating IP) can mount NFS (maps)
  17. VM (floating IP) can mount NFS (maps)
  18. VM (no floating IP) can mount NFS (tools)
  19. VM (floating IP) can mount NFS (tools)
  20. VM (no floating IP) can connect to wiki-replicas
  21. VM (floating IP) can connect to wiki-replicas
  22. VM (no floating IP) can use openstack endpoints
  23. VM (floating IP) can use openstack endpoints
  24. All VMs can use the internal openstack metadata address
  25. puppetmasters VM work as expected
  26. security groups work as expected

stage 2: enable L3 routing on cloudsw nodes

Given we don't have hardware for testing the cloudsw setup in codfw, we assume we are working with core routers and asw.

Therefore this stage is a NOOP in the codfw datacenter.

stage 3: enable L3 routing on cloudgw nodes

All the codfw changes were done in the stage 1, therefore this stage is a NOOP in the codfw datacenter.