Out-of-band network

All of our sites have a dedicated Out Of Band (OOB) network, consisting of internet access (DIA), routers and switches physically separated from our production network.

This OOB access is our last resort solution in case of a site becoming partially (eg. bastions) or fully (eg. transport/transits) unreachable by normal means.

Note that in the event of a site losing connectivity to the internet, it's still reachable through its transport links from any other bastion host.

Prerequisites

To use our OOB network you need to have:

1. An account on the network devices, see this list
2. The password store checked out on your local machine

Step by step

The entry points to SSH to our management routers over their DIA are:

Sites OOB

site	hostname
eqiad	mr1-eqiad.oob.wikimedia.org
codfw	mr1-codfw.oob.wikimedia.org
esams	mr1-esams.oob.wikimedia.org
ulsfo	mr1-ulsfo.oob.wikimedia.org
eqsin	mr1-eqsin.oob.wikimedia.org

From there you can SSH directly to relevant devices on the OOB network, for example a console server, eg:

mr1-eqiad> ssh root@scs-a8-eqiad.mgmt.eqiad.wmnet

Where you will need the matching password from the password store.

Note that it's also possible do to agent forwarding and port forwarding on the management routers to reach devices with password authentication turned off.

No DNS

If DNS resolution doesn't work (for internal or external hosts), you can replace the FQDNs with the IPs listed in our Puppet configuration (OOB, scs, routers, etc)

To protect against the eventuality of being unable to resolve wikitech-static.wikimedia.org to reach this page, you can save a copy of this page locally or look it up in the Wayback Machine. In both cases, there is of course a risk that the last saved version is outdated; also, you would have to know about these workarounds on your own, being unable to look them up here on Wikitech.

See also

• tunnelencabulator - alternate-PoP IP address ssh tunnel management tool