Out-of-band network

All of our sites have a dedicated Out Of Band (OOB) network, consisting of internet access (DIA), routers and switches physically separated from our production network.

This OOB access is our last resort solution in case of a site becoming partially (eg. bastions) or fully (eg. transport/transits) unreachable by normal means.

Note that in the event of a site losing connectivity to the internet, it's still reachable through its transport links from any other bastion host.

Prerequisites

To use our OOB network you need to have:

1. An account on the network devices, see this list
2. The password store checked out on your local machine

Step by step

The entry points to SSH to our management routers over their DIA are:

Sites OOB

site	hostname
eqiad	mr1-eqiad.oob.wikimedia.org
codfw	mr1-codfw.oob.wikimedia.org
esams	mr1-esams.oob.wikimedia.org
ulsfo	mr1-ulsfo.oob.wikimedia.org
eqsin	mr1-eqsin.oob.wikimedia.org
drmrs	mr1-drmrs.oob.wikimedia.org

NOTE: You should make sure that your SSH config does not have any entries that will try to connect to the above hostnames via one of our bastion hosts. To ensure your connection cannot be affected by any in-band changes to the network you need to ensure it goes directly from your machine to the OOB link over the internet.

From there you can SSH directly to relevant devices on the OOB network, for example a console server, eg:

mr1-eqiad> ssh root@scs-a8-eqiad.mgmt.eqiad.wmnet

Where you will need the matching password from the password store.

Note that it's also possible do to agent forwarding and port forwarding on the management routers to reach devices with password authentication turned off.

SSH Algorithms

On some of the more recent JunOS versions you may find the management router cannot connect to older devices, due to the support algorithms/ciphers supported on the destination device not being allowed by default:

cmooney@mr1-eqiad> ssh root@scs-f8-eqiad.mgmt.eqiad.wmnet 
Unable to negotiate with 10.65.0.4 port 22: no matching MAC found. Their offer: umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com

To work around this you can drop to a csh shell and use the regular ssh client, specifying the correct parameters manually:

cmooney@mr1-eqiad> start shell 
% 
% ssh -m hmac-sha2-512-etm@openssh.com root@scs-f8-eqiad.mgmt.eqiad.wmnet
Password: 
# 

No DNS

If DNS resolution doesn't work (for internal or external hosts), you can replace the FQDNs with the IPs listed in our Puppet configuration (OOB, scs, routers, etc)

To protect against the eventuality of being unable to resolve wikitech-static.wikimedia.org to reach this page, you can save a copy of this page locally or look it up in the Wayback Machine. In both cases, there is of course a risk that the last saved version is outdated; also, you would have to know about these workarounds on your own, being unable to look them up here on Wikitech.

See also

• tunnelencabulator - alternate-PoP IP address ssh tunnel management tool