You are browsing a read-only backup copy of Wikitech. The live site can be found at

Portal:Cloud VPS/Infrastructure

From Wikitech-static
< Portal:Cloud VPS
Revision as of 23:25, 29 July 2017 by imported>BryanDavis (Update some things for rebranding)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Cloud VPS is a virtualization cloud that uses OpenStack Compute. Base images are managed with Glance and authentication uses LDAP-backed Keystone.

Cloud VPS currently runs in a single datacenter in Ashburn, Virginia. In the future it will span two or more datacenters, with a slightly different configuration in each.

For troubleshooting immediate labs issues, visit Portal:Cloud_VPS/Admin/Troubleshooting.

File:Labs architecture.pdf

Cloud VPS Eqiad (Ashburn, VA)

The servers that make Wikimedia Labs


In addition to hosting the WMF's technical documentation, the wikitech web site also runs the OpenStackManager MediaWiki extension that provides a graphical interface for labs. Wikitech runs on a server internally named Silver.

An alternative, partially-functional labs GUI can be accessed at It runs the openstack-dashboard project and provides nova API access to project admins.


The labs controller box (currently named 'labcontrol1001') runs the Glance and Keystone services, as well as a few nova services (conductor and scheduler.) Labcontrol1001 also runs a public DNS server (aka labs-ns0) which provides name services for the domain.

A second server, labcontrol1002, serves as a hot spare for labcontrol1001.

Another duplicate server, labcontrol2001, runs in codfw and contains a duplicate config. It is largely vestigial, but does provide backup DNS service via the labs-ns1 service name.


The network node ('labnet1001') hosts the nova-network service. We currently run a single labs-wide network that supports all lab nodes and projects. In the future we hope to use Openstack Neutron for our network setup, but it doesn't support our use-case; to use neutron we'll need to switch to one network per project.

Labnet1001 also hosts the nova-api service.

Soon an additional server, labnet1002, will provide either redundant network service or as a hot spare. To be determined.


The servers that a labs instance talks to

There are currently thirteen virtualization nodes in labs, named labvirt1001-1013, all running in eqiad:

  • 1001-1009 are high-powered multi-CPU HP servers; each of them hosts dozens of virtual machines.
  • 1010-1013 are similar to 1001-1009 but with large SSD raids.
  • 1014 is identical to 1012 and 1013 but kept empty as an emergency evacuation node.


Labs uses shared storage for several purposes:

  • Each member of a project has a project-wide shared home directory.
  • Each project has a public shared volume, generally mounted to /data/project

All of the above are hosted on an NFS server named labstore1001. There's a hot-swappable backup, labstore1002, which is generally turned off.


Labmon1001 runs statsd and graphite. It monitors the state of labs instances and collects stats and sends alerts as needed.


Ldap is used for services throughout the WMF. The primary ldap server is Neptunium, running in eqiad. The secondary server is Nembus, running in codfw.

The LDAP server software is opendj. Each labs instance has an /etc/ldap.conf file (managed by puppet) that maintains info about the ldap servers.


DNS is handled by PowerDNS. Private DNS entries (e.g. foo.eqiad.wmflabs) are created via Designate Sink and stored in a PDNS server using a mysql backend. Public DNS entries are created via Horizon and the designate API.

Future, simpler Labs DNS implementation using Horizon