All of our sites have a dedicated Out Of Band (OOB) network, consisting of internet access (DIA), routers and switches physically separated from our production network.
This OOB access is our last resort solution in case of a site becoming partially (eg. bastions) or fully (eg. transport/transits) unreachable by normal means.
Note that in the event of a site losing connectivity to the internet, it's still reachable through its transport links from any other bastion host.
To use our OOB network you need to have:
- An account on the network devices, see this list
- The password store checked out on your local machine
Step by step
The entry points to SSH to our management routers over their DIA are:
From there you can SSH directly to relevant devices on the OOB network, for example a console server, eg:
mr1-eqiad> ssh email@example.com
Where you will need the matching password from the password store.
Note that it's also possible do to agent forwarding and port forwarding on the management routers to reach devices with password authentication turned off.
If DNS resolution doesn't work (for internal or external hosts), you can replace the FQDNs with the IPs listed in our Puppet configuration (OOB, scs, routers, etc)
To protect against the eventuality of being unable to resolve wikitech-static.wikimedia.org to reach this page, you can save a copy of this page locally or look it up in the Wayback Machine. In both cases, there is of course a risk that the last saved version is outdated; also, you would have to know about these workarounds on your own, being unable to look them up here on Wikitech.
- tunnelencabulator - alternate-PoP IP address ssh tunnel management tool