You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Ganeti

From Wikitech-static
Revision as of 13:21, 16 June 2015 by imported>Alexandros Kosiaris (→‎Cluster upgrades)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Introduction

Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. It supports both KVM and Xen. In WMF we only have KVM as an enabled hypervisor. Primary Ganeti web page is [[1]].

In WMF ganeti is used as a cluster management tool for a private VPS cloud installation. After an evaluation process of Openstack vs Ganeti, Ganeti was chosen as a more fitting software for the job at hand.

Architecture

Overview

Ganeti is architected as a shared nothing cluster with job management. There is one master node that receives all jobs to be executed (create a VM, delete a VM, stop/start VMs, etc) that can be swapped between a preconfigured number of master candidates in case of a hardware failure. That allows for no single point of failure for cluster operations. For VMs operations, provided the DRBD backend is used, which we do in WMF, even in the case of catastrophic failure for a hardware node, VMs can be restarted with minimal disruption on their secondary (backup) node.

A high level overview of the architecture is here [[2]] and more discussion about it is in [[3]]

Specifics

Cluster Operations

Init the cluster

An example of a initializing a new cluster:

  sudo gnt-cluster init \
  --no-ssh-init \ 
  --enabled-hypervisors=kvm \
  --vg-name=ganeti \
  --master-netdev=br0 \
  --hypervisor-parameters kvm:kvm_path=/usr/bin/qemu-system-x86_64,kvm_flag=enabled,serial_speed=115200,kernel_path= \
  --nic-parameters=link=br0 \ 
  ganeti01.svc.codfw.wmnet

The above is the way we currently have our clusters configured

Modify the cluster

Modifying the cluster to change defaults, parameters of hypervisors, limits, security model etc is possibly. An example of modifying the cluster is given below.

  sudo gnt-cluster modify -H kvm:kvm_path=/usr/bin/qemu-system-x86_64,kvm_flag=enabled,kernel_path=

To get an idea of what is actually modifiable do a:

  sudo gnt-cluster info

and then lookup in ganeti documentation the various options [[4]]

Destroy the cluster

Destroying the cluster is a one way street. Do not do it lightly. An example of destroying a cluster:

 sudo gnt-cluster destroy  --yes-do-it

do note that various things will be left behind. For example /var/lib/ganeti/queue/ will not be deleted. It's up to you if you want to delete it or not, depending on the case.

Add a node

Adding a new hardware node to the cluster to increase capacity

  sudo gnt-node add ganeti1002.eqiad.wmnet

Listing cluster nodes

Listing all hardware nodes in a cluster:

  sudo gnt-node list

That should return something like:

  Node                   DTotal  DFree MTotal MNode MFree Pinst Sinst
  ganeti1001.eqiad.wmnet 427.9G 427.9G  63.0G  391M 62.4G     0     0
  ganeti1002.eqiad.wmnet 427.9G 427.9G  63.0G  289M 62.5G     0     0
  ganeti1003.eqiad.wmnet 427.9G 427.9G  63.0G  288M 62.5G     0     0
  ganeti1004.eqiad.wmnet 427.9G 427.9G  63.0G  288M 62.5G     0     0

The columns are respectively: Disk Total, Disk Free, Memory Toal, Memory used by node itself, Memory Free, Instances for which this node is primary, instances for which this node is secondary


View the job queue

Ganeti has a job queue built-in. Most of the times it's working fine but if something is taking too long it might be helpful to check what's going on in the job queue

  gnt-job list

and getting a job id from the result

  gnt-job info #ID

Cluster upgrades

Hardware/software upgrades on a ganeti cluster can happen with 0 downtime to the VMs operations. The procedure to do so is outlined below. In case a shutdown/reboot is needed the procedure to empty to node is described. The rolling

Do the software upgrade (if needed)

  apt-get safe-upgrade

throughout the cluster. It should have 0 repercussions to any VM anyway. Barring a Ganeti bug in the upgraded version, the cluster itself should also have 0 problems

Rolling reboot

Doing a rolling reboot of the cluster is easy. Empty every node, reboot it, check that it is online, proceed to the next. The one thing to take care is to not reboot the master without failing it over first.

Failover the master

Choose a master candidate that suits you. You can get master candidates by

  sudo gnt-node list -o name,master_candidate

Login and

  sudo gnt-cluster master-failover

The cluster IP will now be served by the new node and the old one is no longer the master.

Node operations

Reboot/Shutdown for maintenance a node

Select a node that needs rebooting/shutdown for brief hardware maintenance and empty of primary instances

   sudo gnt-node migrate -f ganeti1004.eqiad.wmnet

Now a

   sudo gnt-node list

should return 0 primary instances for the node. It is safe to reboot it or shut it down for a brief amount of time for hardware maintenance

Shutdown a node for a prolonged period of time

Should the node be going down for an undetermined amount of time, also move the secondary instances

   sudo gnt-node migrate -f <node_fqdn>
   sudo gnt-node evacuate -s  <node_fqdn>

The second command means moving around DRBD pairs and syncing disk data. It is bound to take a long time, so find something else to do in the meanwhile

Now a

   sudo gnt-node list 

should return 0 for both primary instances as well as secondary instances. Before poweroff the node we need to remove it from the cluster as well

   sudo gnt-node remove <node_fqdn>

NOTE: Do not forget to readd it after it is fixed (if it ever is)

   sudo gnt-node add <node_fqdn>

VM operations

Listing VMs

   gnt-instance list 

Create a VM

Creating a VM is easy. Most of the steps are the same as for production so keep in mind the regular process as well.

Assign a hostname/IP

Same process as for hardware. Assign the IP/hostname and make sure DNS changes are live before going forward.

Create the VM

   gnt-instance add \
     -t drbd \
     -I hail \
     --net 0:link=br0 \
     --hypervisor-parameters=kvm:boot_order=network \
     -o debootstrap+default \
     --no-install \
     -B vcpus=<x>,memory=<y>g --disk 0:size=<z>g \
     <fqdn>

Note the the VM will NOT be started. That's on purpose for now. x, y ,z on the above are variables. t,g,m denote tera,giga,mega bytes respectively.

Get the MAC address of the NIC

   gnt-instance info <fqdn> | grep MAC

Get the MAC address

Update DHCP

Same as usual. Use linux-host-entries.ttyS0-115200 for Ganeti VMs. Otherwise you will not be getting a console

Update autoinstall files

Same as usual. Do however add virtual.cfg to the configuration for a specific VM. Example: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/install-server/files/autoinstall/netboot.cfg;601720be51228f7eae3de17988b1afa8881a5bdb$71

Start the VM

  gnt-instance start <fqdn>

and connect to the console

  gnt-instance console <fqdn>

Ctrl+] to leave the console

Set boot order to disk

Assuming the installation goes on well but before it finishes, you need to set the boot order back to disk. This is a limitation of the current version of the Ganeti software and is expected to be solved (upstream is aware).

  gnt-instance modify \
  --hypervisor-parameters=boot_order=disk \
  <fqdn>

Note: when the VM has finished installing, it will shutdown automatically. The Ganeti software includes HA checks and will promptly restart it. We rely on this behaviour to have the VM successfully installed. However, if you list the VMs during this phase you will see the VM in ERROR_down. Don't worry, this is expected.

Assign role to the VM in puppet

As usual.

Delete a VM

Irrevocably deleting a VM is done via:

  gnt-instance remove <fqdn>

Please remember to clean up DHCP/DNS entries afterwards

Shutdown/startup a VM

  gnt-instance startup <fqdn>
  gnt-instance shutdown <fqdn>

Note: In the shutdown command, ACPI will be used to achieve a graceful shutdown of the VM. A 2 minute timeout exists however, after which the VM will be forcefully shutdown. In case you prefer to not wait those 2 minutes, --timeout exists and can be used like so

  gnt-instance shutdown --timeout 0 <fqdn>

Notes

All of the commands that have a Y/N prompt can be forced with a -f. For example the following will spare you the prompt

  gnt-instance remove -f <fqdn>

All commands are actually jobs. If you would rather not wait on the prompt --submit will do the trick

  gnt-instance shutdown --submit <fqdn>

API