You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Portal:Cloud VPS/Admin/VM images

From Wikitech-static
< Portal:Cloud VPS‎ | Admin
Revision as of 22:54, 24 August 2018 by imported>Andrew Bogott (→‎Install a new image: Updated labcontrol1001 references to cloudcontrol1003)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

This page collects all the information on VM images for instances that runs on Cloud VPS.

Image generation

There are VM images out there (Debian, Ubuntu) which are cloud-ready but we do generate our owns.

Debian

We build debian images using bootstrap-vz. The bootstrap-vz config is puppetized in the class labs_bootstrapvz -- on Jessie we use a custom build of the bootstrap-vz package, documented below.

To build a Debian Jessie image, log in to labs-bootstrapvz-jessie:

sudo su -
cd /target # This is where the image will end up when we finish
rm *.raw && rm *.qcow2 # Make space for our new build
bootstrap-vz /etc/bootstrap-vz/manifests/labs-jessie.manifest.yaml
qemu-img convert -f raw -O qcow2 ./debian-jessie.raw ./debian-jessie.qcow2

Custom bootstrap-vz package

Andrew built python-bootstrap-vz_0.9wmf-1_all.deb using stddeb. It was built from the 'development' branch on 2014-12-22 with commit 255f0624b49dbcf6cacccd3b2f1fa7c0cc2bcc8d and the patch, below. To reproduce:

diff --git a/setup.py b/setup.py
index f7b97ac..349cfdc 100644
--- a/setup.py
+++ b/setup.py
@@ -22,11 +22,8 @@ setup(name='bootstrap-vz',
       install_requires=['termcolor >= 1.1.0',
                         'fysom >= 1.0.15',
                         'jsonschema >= 2.3.0',
-                        'pyyaml >= 3.10',
                         'boto >= 2.14.0',
                         'docopt >= 0.6.1',
-                        'pyrfc3339 >= 1.0',
-                        'requests >= 2.9.1',
                         ],
       license='Apache License, Version 2.0',
       description='Bootstrap Debian images for virtualized environments',


  • Alter the version tag in vi bootstrapvz/__init__.py as needed
  • Install python-stdeb
  • python setup.py --command-packages=stdeb.command bdist_deb
  • ls deb_dist/*.deb

As of 2017-05-06, the .deb provided by the upstream debian Stretch repo (bootstrap-vz 0.9.10+20170110git-1) seems to work properly on Stretch without a custom build or any additional modifications.

Ubuntu

We use vmbuilder to build our custom Ubuntu images. The vmbuilder configuration is in puppet in the labs-vmbuilder module. It can be added to a node using role::labs::vmbuilder. Here's a set of steps to build and import the images:

On vmbuilder-trusty.openstack.eqiad.wmflabs:

puppet agent -tv
cd /srv/vmbuilder
rm -Rf ubuntu-trusty
vmbuilder kvm ubuntu -c /etc/vmbuilder.cfg -d /srv/vmbuilder/ubuntu-trusty -t /srv/vmbuilder/tmp --part=/etc/vmbuilder/files/vmbuilder.partition

Note the name of the tmp file generated; for instance: "Converting /tmp/tmpD0yIQa to qcow2, format /mnt/vmbuilder/ubuntu-trusty/tmpD0yIQa.qcow2"

Note the name of the tmp file generated; for instance: "Converting /tmp/tmpD0yIQa to qcow2, format /mnt/vmbuilder/ubuntu-trusty/tmpD0yIQa.qcow2"

Image management

The final destination of VM images are the Openstack Glance service.

Image test/debug

You can boot an image locally for testing, like this:

qemu-system-x86_64 ./<new image name>.raw --curses

Unfortunately, qemu's default behavior is to suppress all boot logs, so you'll be looking at a mostly-blank screen for several minutes before getting a login prompt with no working password. Turning on a working login account for test purposes is left as an exercise to the reader -- bootstrap-vz creates one by default (login: root passwd:test) but our config wisely disables it.

Bootstrap-vz uses source files from /etc/bootstrap-vz. These files are puppetized, so you'll want to disable puppet if you change them.

Bootstrap-vz also uses several source files that are standard local config files on the build host. For a complete list of these files, look at the 'file_copy:' section in /etc/bootstrap-vz/manifests/labs-jessie.manifest.yaml

The build process for Stretch is similar; the build system is labs-bootstrapvz-stretch.openstack.eqiad.wmflabs and the manifest to use is named labs-stretch.manifest.yaml.

Another method is to run:

sudo qemu-system-x86_64 -nographic -serial mon:stdio -enable-kvm ./debian-jessie.raw

This way you will see all the boot prompt.

Install a new image

On cloudcontrol1003 (or other control box):

First, get the new .qcow2 images into /tmp on cloudcontrol1003.wikimedia.org. One way is to rsync it from the builder VM to your laptop using your labs/labs root key, and rsync it to cloudcontrol1003 using your prod key. For example:

rsync -e 'ssh -i ~/.ssh/id_rsa_labs_private' root@vmbuilder-trusty.openstack.eqiad.wmflabs:/srv/vmbuilder/ubuntu-trusty/tmpUUpMh4.qcow2 .
rsync -e 'ssh -i ~/.ssh/id_rsa_prod_new' tmpUUpMh4.qcow2 madhuvishy@cloudcontrol1003.wikimedia.org:/tmp/

Then, also on cloudcontrol1003...

sudo su -
source ~/novaenv.sh 
cd /tmp
openstack image create --file ubuntu-trusty.qcow2 --disk-format "qcow2" --container-format "ovf" --public "ubuntu-14.04-trusty (testing)"
openstack image create --file debian-jessie.qcow2 --disk-format "qcow2" --container-format "ovf" --public "debian-8.1-jessie (testing)"
# Test the images by booting instances in labs; if they don't work, delete
# the instances, then delete the images (using glance delete), then
# restart the process 
glance index # image-list in mitaka
# find image ids
glance image-update --name "ubuntu-14.04-trusty (deprecated <date>)" <old-trusty-id> --purge-props
glance image-update --name "ubuntu-14.04-trusty" <new-trusty-id> --property show=true --property default=true
glance image-update --name "ubuntu-12.04-precise (deprecated <date>)" <old-precise-id> --purge-props
glance image-update --name "ubuntu-12.04-precise" <new-precise-id> --property show=true

Notice in the above glance image-update commands the use of properties. If default=true the image will be the default image selected in the instance creation interface; purging properties removes the 'default' state.

We used to use the 'show=true' property to ensure that an image appeared on wikitech. Now instead we use the image state, where only images with state=active appear in the gui (both on wikitech and in Horizon.) To deactivate your obsolete image:

source /etc/novaenv.sh 
openstack token issue
curl -X POST -H 'X-Auth-Token: <token id>' http://cloudcontrol1003.wikimedia.org:9292/v2/images/<image id>/actions/deactivate

To reactivate an image (because it was deactivated in error, or in order to permit a migration):

source /etc/novaenv.sh 
openstack token issue
curl -X POST -H 'X-Auth-Token: <token id>' http://cloudcontrol1003.wikimedia.org:9292/v2/images/<image id>/actions/reactivate

Delete an old image

TODO: fill me.

First boot

The first boot of the VM image is a key moment in the setup of an instance in Cloud VPS.

This is usually done by means of the /root/firstboot.sh script which is called by means of /etc/rc.local.

The script will do:

  • some LVM configuration
  • run DHCP request for configuration
  • name/domain resolution to autoconfigure the VM
  • initial puppet autoconfiguration (cert request, etc)
  • initial configuration of nscd/nslcd
  • initial apt updates
  • NFS mounts if required
  • final puppet run to fetch all remaining configuration (ssh keys, packages, etc)

Until the last point, the instance may have limited connectivity or usability.

Common issues and hacks

These problems may vary from deployment to deployment (main, labtest, labtestn), but they could be common.

  • Image does not have the puppet master CA, so it fails to fetch catalog (see phab:T181523)
  • Image does not have the puppet master CRL, so it fails to fetch catalog (see phab:T181523)
  • Image doesn't correctly resolve the hostname/domain name (so it fails to fetch is own puppet catalog)

Inspecting a disk by hand

If you want to explore & edit the disk image of a live instance, follow this procedure:

  • locate the virt server in which the VM is running
aborrero@labtestvirt2001:~ $ for i in $(sudo virsh list --all | grep i.* | awk -F' ' '{print $2}') ; do echo -n "$i " ; sudo virsh dumpxml $i | grep nova:name ; done
i-0000025d       <nova:name>puppettestingui</nova:name>
i-00000262       <nova:name>aptproxy2</nova:name>
i-00000263       <nova:name>t2</nova:name>
i-00000264       <nova:name>t3</nova:name>
  • Once you know the internal instance name (i-xxxxx), locate the disk file
aborrero@labtestvirt2001:~ $ sudo virsh dumpxml i-00000264 | grep "source file" | grep disk
      <source file='/var/lib/nova/instances/09865310-b440-4dc7-99ab-fb5f35be04fb/disk'/>
  • shutdown the machine (from inside, from horizon, using virsh or whatever)
  • copy the disk file to your home
aborrero@labtestvirt2001:~ $ cp /var/lib/nova/instances/09865310-b440-4dc7-99ab-fb5f35be04fb/disk t3-disk.qcow2
  • disable puppet and install libguestfs-tools
aborrero@labtestvirt2001:~ $ sudo puppet agent --disable "inspecting instance disk" ; sudo aptitude install libguestfs-tools -y
[...]
  • create a destination directory and mount the disk!
aborrero@labtestvirt2001:~ $ mkdir mnt ; sudo guestmount -a t3-disk.qcow2 -m /dev/sda3 --rw mnt
  • you can now read/write the instance disk in the mount point
  • when done, umount, copy back the instance disk and start the instance!

See also