Ganeti

Introduction

Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. It supports both KVM and Xen. In WMF we only have KVM as an enabled hypervisor. Primary Ganeti web page is [[1]].

In WMF ganeti is used as a cluster management tool for a private VPS cloud installation. After an evaluation process of Openstack vs Ganeti, Ganeti was chosen as a more fitting software for the job at hand.

Architecture

Overview

Ganeti is architected as a shared nothing cluster with job management. There is one master node that receives all jobs to be executed (create a VM, delete a VM, stop/start VMs, etc) that can be swapped between a preconfigured number of master candidates in case of a hardware failure. That allows for no single point of failure for cluster operations. For VMs operations, provided the DRBD backend is used, which we do in WMF, even in the case of catastrophic failure for a hardware node, VMs can be restarted with minimal disruption on their secondary (backup) node.

A high level overview of the architecture is here [[2]] and more discussion about it is in [[3]]

Specifics

A cluster is identified by:

• The nodes
• An FQDN (e.g ganeti01.svc.eqiad.wmnet), which obviously corresponds to an IPv4 address. That IPv4 address is "floating", meaning that it is owned by the current master.

Administration always happens via the master. It is the only node where all commands can be run and hosts the API. Failover of a master is easy but manual. See below for more information on how to do it.

Connect to a cluster

Just ssh to its FQDN

  ssh ganeti01.svc.eqiad.wmnet

Cluster Operations

Init the cluster

An example of a initializing a new cluster:

  sudo gnt-cluster init \
  --no-ssh-init \ 
  --enabled-hypervisors=kvm \
  --vg-name=ganeti \
  --master-netdev=br0 \
  --hypervisor-parameters kvm:kvm_path=/usr/bin/qemu-system-x86_64,kvm_flag=enabled,serial_speed=115200,kernel_path= \
  --nic-parameters=link=br0 \ 
  ganeti01.svc.codfw.wmnet

The above is the way we currently have our clusters configured

Modify the cluster

Modifying the cluster to change defaults, parameters of hypervisors, limits, security model etc is possibly. An example of modifying the cluster is given below.

  sudo gnt-cluster modify -H kvm:kvm_path=/usr/bin/qemu-system-x86_64,kvm_flag=enabled,kernel_path=

To get an idea of what is actually modifiable do a:

  sudo gnt-cluster info

and then lookup in ganeti documentation the various options [[4]]

Destroy the cluster

Destroying the cluster is a one way street. Do not do it lightly. An example of destroying a cluster:

 sudo gnt-cluster destroy  --yes-do-it

do note that various things will be left behind. For example /var/lib/ganeti/queue/ will not be deleted. It's up to you if you want to delete it or not, depending on the case.

Add a node

Adding a new hardware node to the cluster to increase capacity

  sudo gnt-node add ganeti1002.eqiad.wmnet

Listing cluster nodes

Listing all hardware nodes in a cluster:

  sudo gnt-node list

That should return something like:

  Node                   DTotal  DFree MTotal MNode MFree Pinst Sinst
  ganeti1001.eqiad.wmnet 427.9G 427.9G  63.0G  391M 62.4G     0     0
  ganeti1002.eqiad.wmnet 427.9G 427.9G  63.0G  289M 62.5G     0     0
  ganeti1003.eqiad.wmnet 427.9G 427.9G  63.0G  288M 62.5G     0     0
  ganeti1004.eqiad.wmnet 427.9G 427.9G  63.0G  288M 62.5G     0     0

The columns are respectively: Disk Total, Disk Free, Memory Toal, Memory used by node itself, Memory Free, Instances for which this node is primary, instances for which this node is secondary

View the job queue

Ganeti has a job queue built-in. Most of the times it's working fine but if something is taking too long it might be helpful to check what's going on in the job queue

  gnt-job list

and getting a job id from the result

  gnt-job info #ID

Cluster upgrades

Hardware/software upgrades on a ganeti cluster can happen with 0 downtime to the VMs operations. The procedure to do so is outlined below. In case a shutdown/reboot is needed the procedure to empty to node is described. The rolling

Do the software upgrade (if needed)

  apt-get safe-upgrade

throughout the cluster. It should have 0 repercussions to any VM anyway. Barring a Ganeti bug in the upgraded version, the cluster itself should also have 0 problems

Rolling reboot

Doing a rolling reboot of the cluster is easy. Empty every node, reboot it, check that it is online, proceed to the next. The one thing to take care is to not reboot the master without failing it over first.

Failover the master

Choose a master candidate that suits you. You can get master candidates by

  sudo gnt-node list -o name,master_candidate

Login and

  sudo gnt-cluster master-failover

The cluster IP will now be served by the new node and the old one is no longer the master.

Node operations

Reboot/Shutdown for maintenance a node

Select a node that needs rebooting/shutdown for brief hardware maintenance and empty of primary instances

   sudo gnt-node migrate -f ganeti1004.eqiad.wmnet

Now a

   sudo gnt-node list

should return 0 primary instances for the node. It is safe to reboot it or shut it down for a brief amount of time for hardware maintenance

Shutdown a node for a prolonged period of time

Should the node be going down for an undetermined amount of time, also move the secondary instances

   sudo gnt-node migrate -f <node_fqdn>
   sudo gnt-node evacuate -s  <node_fqdn>

The second command means moving around DRBD pairs and syncing disk data. It is bound to take a long time, so find something else to do in the meanwhile

Now a

   sudo gnt-node list 

should return 0 for both primary instances as well as secondary instances. Before poweroff the node we need to remove it from the cluster as well

   sudo gnt-node remove <node_fqdn>

NOTE: Do not forget to readd it after it is fixed (if it ever is)

   sudo gnt-node add <node_fqdn>

VM operations

Listing VMs

   gnt-instance list 

Create a VM

Creating a VM is easy. Most of the steps are the same as for production so keep in mind the regular process as well.

Assign a hostname/IP

Same process as for hardware. Assign the IP/hostname and make sure DNS changes are live before going forward.

Create the VM (private IP)

   gnt-instance add \
     -t drbd \
     -I hail \
     --net 0:link=br0 \
     --hypervisor-parameters=kvm:boot_order=network \
     -o debootstrap+default \
     --no-install \
     -B vcpus=<x>,memory=<y>g \
     --disk 0:size=<z>g \
     <fqdn>

Note the the VM will NOT be started. That's on purpose for now. x, y ,z on the above are variables. t,g,m denote tera,giga,mega bytes respectively.

Create the VM (public IP)

   gnt-instance add \
     -t drbd \
     -I hail \
     --net 0:link=br0.<VLAN_ID> \
     --hypervisor-parameters=kvm:boot_order=network \
     -o debootstrap+default \
     --no-install \
     -B vcpus=<x>,memory=<y>g \
     --disk 0:size=<z>g \
     <fqdn>

Note that the only difference between public/private IP is the <VLAN_ID> required. This is the VLAN ID used by the network. Here's multiple different ways to find it out:

• The easiest (USE THIS FOR NOW) Use the ganeti nodes themselves: /etc/network/interfaces. All support VLANS apart from the very basic one (the one the ganeti hosts are installed into anyway are there in the form of br0.<VLAN_ID>. For now it is just one as we don't yet support cross row.
• Use the switches: This requires knowning just a bit more about the network. Figure out the row the ganeti hosts are racked into, log in into the corresponding switch and get the vlan ID with show vlans <id_name>.

Get the MAC address of the NIC

   gnt-instance info <fqdn> | grep MAC

Get the MAC address

Update DHCP

Same as usual. Use linux-host-entries.ttyS0-115200 for Ganeti VMs. Otherwise you will not be getting a console

Update autoinstall files

Same as usual. Do however add virtual.cfg to the configuration for a specific VM. Example: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/install-server/files/autoinstall/netboot.cfg;601720be51228f7eae3de17988b1afa8881a5bdb$71

Start the VM

  gnt-instance start <fqdn>

and connect to the console

  gnt-instance console <fqdn>

Ctrl+] to leave the console

Set boot order to disk

Assuming the installation goes on well but before it finishes, you need to set the boot order back to disk. This is a limitation of the current version of the Ganeti software and is expected to be solved (upstream is aware).

  gnt-instance modify \
  --hypervisor-parameters=boot_order=disk \
  <fqdn>

Note: when the VM has finished installing, it will shutdown automatically. The Ganeti software includes HA checks and will promptly restart it. We rely on this behaviour to have the VM successfully installed. However, if you list the VMs during this phase you will see the VM in ERROR_down. Don't worry, this is expected.

Assign role to the VM in puppet

As usual.

Delete a VM

Irrevocably deleting a VM is done via:

  gnt-instance remove <fqdn>

Please remember to clean up DHCP/DNS entries afterwards

Shutdown/startup a VM

  gnt-instance startup <fqdn>
  gnt-instance shutdown <fqdn>

Note: In the shutdown command, ACPI will be used to achieve a graceful shutdown of the VM. A 2 minute timeout exists however, after which the VM will be forcefully shutdown. In case you prefer to not wait those 2 minutes, --timeout exists and can be used like so

  gnt-instance shutdown --timeout 0 <fqdn>

Notes

All of the commands that have a Y/N prompt can be forced with a -f. For example the following will spare you the prompt

  gnt-instance remove -f <fqdn>

All commands are actually jobs. If you would rather not wait on the prompt --submit will do the trick

  gnt-instance shutdown --submit <fqdn>