You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Portal:Cloud VPS/Admin/VM images: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Andrew Bogott
(→‎Install a new image: Updated labcontrol1001 references to cloudcontrol1003)
 
imported>Quiddity
m (fixes)
 
(13 intermediate revisions by 7 users not shown)
Line 1: Line 1:
{{draft}}
Cloud VPS uses special VM images that contain all the customizations required for our environment.


This page collects all the information on '''VM images''' for instances that runs on [[Portal:Cloud_VPS | Cloud VPS]].
== Builders ==


= Image generation =
Images for Debian Buster (and subsequent release) are based off of official debian upstream builds. These are built on a cloudcontrol node.


There are VM images out there (Debian, Ubuntu) which are ''cloud-ready'' but we do generate our owns.
Due to an old version of cloud-init that is bundled with upstream Stretch, we build custom base images (with modern cloud-init) for Stretch. Those are built on with bootstrap-vz on cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs.


== Debian ==
== Building Pre-puppetized Images with wmcs-image-create ==
We build debian images using bootstrap-vz.  The bootstrap-vz config is puppetized in the class labs_bootstrapvz -- on Jessie we use a custom build of the bootstrap-vz package, documented below.


To build a Debian Jessie image, log in to labs-bootstrapvz-jessie:
Our cloud should work with any upstream Debian base image that includes cloud-init version later than 18.4. The Debian project provides [https://cdimage.debian.org/cdimage/openstack/ ready-made] [https://cloud.debian.org/images/cloud/ images] prepared for use with OpenStack.


<source lang="bash">
For faster startup and better storage performance, we typically use a pre-puppetized snapshot based on an upstream image. The wmcs-image-create script will automatically generate and install a pre-puppetized image. This script is invoked with a url pointing to the desired upstream image to use as a starting point. It boots a temporary VM using that image, puppetized that VM, and creates a new image based on that puppetized VM.
 
<syntaxhighlight lang="bash">
root@cloudcontrol1003:~# wmcs-image-create --image-url https://cdimage.debian.org/cdimage/openstack/10.8.3-20210304/debian-10.8.3-20210304-openstack-amd64.raw --new-image-name "debian-10 example build" --project-owner "testlabs"
Downloading upstream image...
--2021-03-25 19:54:41--  https://cdimage.debian.org/cdimage/openstack/10.8.3-20210304/debian-10.8.3-20210304-openstack-amd64.raw
Resolving cdimage.debian.org (cdimage.debian.org)... 2001:6b0:19::165, 2001:6b0:19::173, 194.71.11.165, ...
Connecting to cdimage.debian.org (cdimage.debian.org)|2001:6b0:19::165|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://saimei.ftp.acc.umu.se/cdimage/openstack/10.8.3-20210304/debian-10.8.3-20210304-openstack-amd64.raw [following]
--2021-03-25 19:54:42--  https://saimei.ftp.acc.umu.se/cdimage/openstack/10.8.3-20210304/debian-10.8.3-20210304-openstack-amd64.raw
Resolving saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)... 2001:6b0:19::138, 194.71.11.138
Connecting to saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)|2001:6b0:19::138|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2147483648 (2.0G)
Saving to: ‘/usr/local/sbin/wmcs-image-createikq36pde/upstreamimage’
 
/usr/local/sbin/wmcs-image-createikq36 100%[===========================================================================>]  2.00G  64.8MB/s    in 33s   
 
2021-03-25 19:55:16 (61.5 MB/s) - ‘/usr/local/sbin/wmcs-image-createikq36pde/upstreamimage’ saved [2147483648/2147483648]
 
Loading the upstream image into glance
Launching a vm with the new image
Created temporary VM dc7ad6ed-b7d0-4631-b034-630b66ea0450
Sleeping 5 minutes while we wait for the VM to start up...
Waiting one minutes for VM to puppetize
Stopping the VM
Taking a snapshot of the stopped instance
Creating snapshot ec347068-949f-42c9-9ded-000c165b3b50
Waiting for snapshot to finish saving...
Grabbing handle to snapshot data
Downloading snapshot to /usr/local/sbin/wmcs-image-createikq36pde/snapshot.img
Making snapshot file sparse
Creating final image 15cd26bc-15a5-4b8e-b0bb-cc1bd6c5ca24
Setting image ownership and visibility
Cleaning up intermediate VM
Cleaning up VM snapshot
Cleaning up upstream image
Finished creating new image: 15cd26bc-15a5-4b8e-b0bb-cc1bd6c5ca24
</syntaxhighlight>
 
The total run should take 10-15 minutes. Once the new image has been tested it needs to be renamed and marked public so that it's available to all projects:
 
<syntaxhighlight lang="bash">
# note IDs of outgoing and incoming image
root@cloudcontrol1003:~#  openstack image list
# Rename and deactivate the former image
root@cloudcontrol1003:~#  openstack image set --name "debian-10.0-buster (deprecated yyyy-mm-dd)" --deactivate <former active image id>
# Publish the new image
root@cloudcontrol1003:~#  openstack image set --name "debian-10.0-buster (deprecated yyyy-mm-dd)" --public <new image build id>
</syntaxhighlight>
 
== Legacy Build Process for Debian Stretch ==
 
=== Building with bootstrap-vz ===
 
* Login to <code>cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs</code>
 
* Build and convert image
 
<syntaxhighlight lang="bash">
sudo su -
sudo su -
cd /target # This is where the image will end up when we finish
cd /target
rm *.raw && rm *.qcow2 # Make space for our new build
bootstrap-vz --pause-on-error  /etc/bootstrap-vz/manifests/labs-stretch.manifest.yaml
bootstrap-vz /etc/bootstrap-vz/manifests/labs-jessie.manifest.yaml
qemu-img convert -f raw -O qcow2 debian-stretch-amd64-$(date "+%Y%m%d").raw debian-stretch-amd64-$(date "+%Y%m%d").qcow2
qemu-img convert -f raw -O qcow2 ./debian-jessie.raw ./debian-jessie.qcow2
</syntaxhighlight>
</source>


=== Custom bootstrap-vz package ===
==== Local bootstrap-vz for stretch ====


Andrew built python-bootstrap-vz_0.9wmf-1_all.deb using stddeb.  It was built from the 'development' branch on 2014-12-22 with commit 255f0624b49dbcf6cacccd3b2f1fa7c0cc2bcc8d and the patch, below. To reproduce:
The version of bootstrap-vz built for Stretch doesn't support setting mount options in the manifest. Since we want 'discard' set, the local install of bootstrapvz on cloud-bootstrapvz-stretch.openstack.eqiad1.wikimedia.cloud has a hack in place to make this setting, in bootstrapvz/common/tasks/filesystem.py:


* Checkout the bootstrap-vz source from https://github.com/andsens/bootstrap-vz
<syntaxhighlight lang="bash">
* add this patch to make stdeb's dependencies work out properly:
190c190
<            mount_opts = ['defaults']
---
>            mount_opts = ['discard', 'defaults']
</syntaxhighlight>


diff --git a/setup.py b/setup.py
=== How To Test ===
index f7b97ac..349cfdc 100644
--- a/setup.py
+++ b/setup.py
@@ -22,11 +22,8 @@ setup(name='bootstrap-vz',
        install_requires=['termcolor >= 1.1.0',
                          'fysom >= 1.0.15',
                          'jsonschema >= 2.3.0',
-                        'pyyaml >= 3.10',
                          'boto >= 2.14.0',
                          'docopt >= 0.6.1',
-                        'pyrfc3339 >= 1.0',
-                        'requests >= 2.9.1',
                          ],
        license='Apache License, Version 2.0',
        description='Bootstrap Debian images for virtualized environments',


You can boot an image locally for testing, like this:


* Alter the version tag in vi bootstrapvz/__init__.py as needed
<code>
* Install python-stdeb
sudo qemu-system-x86_64 -nographic -serial mon:stdio -enable-kvm image_name.raw
* python setup.py --command-packages=stdeb.command bdist_deb
</code>
* ls deb_dist/*.deb


As of 2017-05-06, the .deb provided by the upstream debian Stretch repo (bootstrap-vz 0.9.10+20170110git-1) seems to work properly on Stretch without a custom build or any additional modifications.
If the command above does not work, you try can try the following command (beware boot logs will be supressed):


== Ubuntu ==
<code>
qemu-system-x86_64 image_name.raw --curses
</code>


{{warning | Usage of Ubuntu is discouraged as it will be deprecated and support disabled at some point in the future by the WMCS team}}
Having a working login account for test purposes is left as an exercise to the reader. bootstrap-vz creates one by default (login:root / passwd:test) but our config wisely disables it.


We use [https://help.ubuntu.com/12.04/serverguide/jeos-and-vmbuilder.html vmbuilder] to build our custom Ubuntu images. The vmbuilder configuration is in puppet in the labs-vmbuilder module.
=== How To Deploy ===
It can be added to a node using role::labs::vmbuilder. Here's a set of steps to build and import the images:


On vmbuilder-trusty.openstack.eqiad.wmflabs:
Images are deployed to the OpenStack Glance service.


<source lang="bash">
* Copy the .qcow2 file to /tmp on the <code>cloudcontrol1003.wikimedia.org</code> server
puppet agent -tv
cd /srv/vmbuilder
rm -Rf ubuntu-trusty
vmbuilder kvm ubuntu -c /etc/vmbuilder.cfg -d /srv/vmbuilder/ubuntu-trusty -t /srv/vmbuilder/tmp --part=/etc/vmbuilder/files/vmbuilder.partition
</source>


Note the name of the tmp file generated; for instance: "Converting /tmp/tmpD0yIQa to qcow2, format /mnt/vmbuilder/ubuntu-trusty/tmpD0yIQa.qcow2"
Since the file has to cross the Cloud VPS / Production boundary, you can copy it from the builder server to your laptop (using your Cloud Services root key) and then from your laptop to cloudcontrol1003 (using your production key):


{{note|On Trusty installing the kernel fails with <tt>This kernel does not support a non-PAE CPU</tt>. The version of python-vmbuilder running on labs-vmbuilder-trusty has a hotfix which is not part of the standard release. This fix is documented [https://bugs.launchpad.net/ubuntu/+source/vm-builder/+bug/1037607 here] and consists of the following diff which needs to be applied manually:
<syntaxhighlight lang="bash">
rsync --progress -v -e ssh root@cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs:/target/debian-stretch-amd64-$(date "+%Y%m%d").qcow2 .
rsync --progress -v -e ssh debian-stretch-amd64-$(date "+%Y%m%d").qcow2 cloudcontrol1003.wikimedia.org:/tmp/
</syntaxhighlight>


<source lang="python">
Alternatively, you can open a temporary HTTP server to make this transfer:
--- /usr/lib/python2.7/dist-packages/VMBuilder/plugins/ubuntu/dapper.py.orig 2012-10-19 15:43:30.149339398 +0300
+++ /usr/lib/python2.7/dist-packages/VMBuilder/plugins/ubuntu/dapper.py      2012-10-19 16:13:45.309289300 +0300
@@ -300,7 +300,9 @@
        return (mirror, updates_mirror, security_mirror)


    def install_kernel(self, destdir):
<syntaxhighlight lang="shell-session>
+        self.run_in_target('mount', '-t', 'proc', 'proc', '/proc')
cloud-bootstrapvz-stretch:~ $ cd /target
        run_cmd('chroot', destdir, 'apt-get', '--force-yes', '-y', 'install', self.kernel_name(), env={ 'DEBIAN_FRONTEND' : 'noninteractive' })
cloud-bootstrapvz-stretch:~ $ python3 -m http.server 80
+        run_cmd('umount', '%s/proc' % self.context.chroot_dir)
</source>


|gotcha}}
cloudcontrol1003:/tmp$ wget http://185.15.56.45/debian-stretch-amd64-$(date "+%Y%m%d").qcow2
</syntaxhighlight>


Note the name of the tmp file generated; for instance: "Converting /tmp/tmpD0yIQa to qcow2, format /mnt/vmbuilder/ubuntu-trusty/tmpD0yIQa.qcow2"
* Login to <code>cloudcontrol1003.wikimedia.org</code>


= Image management =
* Convert the image from qcow2 to raw/sparse.  This is a new step necessary now that images are stored using ceph/rbd:


The final destination of VM images are the Openstack Glance service.
<syntaxhighlight lang="bash">
qemu-img convert -f qcow2 -O raw  ./<filename>.qcow2 ./<filename>.raw.notsparse
cp --sparse=always ./<filename>.raw.notsparse  ./<filename>.raw
</syntaxhighlight>


== Image test/debug ==
* Create new image in Glance:


You can boot an image locally for testing, like this:
{{warning |  Only rename the existing images AFTER you upload the new images. We have [[Nova_Resource:Admin-monitoring|monitoring]] in place that depends on the exact image names. Remember to update it as well (and wait for Puppet to run and update fullstackd). }}
 
<syntaxhighlight lang="bash">
sudo su -
source ~/novaenv.sh
cd /tmp
openstack image create --file debian-stretch-amd64-$(date "+%Y%m%d").raw --disk-format "raw" --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi --container-format "ovf" --public "debian-9.6-stretch"
</syntaxhighlight>
 
* Test new image by booting a new VM with it (if the image is faulty, remember to delete the test VM and the faulty image)
 
* Update fullstackd to use this new image (see T218314 for an example).
 
* Get a list of existing images
 
<syntaxhighlight lang="bash">
openstack image list
</syntaxhighlight>
 
* Append "deprecated" to old images and remove properties (only if new image is working as expected)
 
<syntaxhighlight lang="bash">
openstack image set --name "debian-9.5-stretch (deprecated <date>)" <old-image-id>
nova image-meta <old-image-id> delete default
nova image-meta <old-image-id> delete show
</syntaxhighlight>
 
Passing <code>--purge-props</code> to <code>openstack image set</code> should be enough to clear all properties but it's currently not available in our OpenStack version. The <code>nova image-meta</code> commands serve the same purpose but you have to delete each property individually. This should be reviewed when OpenStack is upgraded.
 
* Make the new image the default for new instances
 
<syntaxhighlight lang="bash">
openstack image set --name "debian-9.6-stretch" <new-image-id> --property show=true --property default=true
</syntaxhighlight>
 
Notice in the above <code>glance image-update</code> commands the use of properties. If <code>default=true</code> the image will be the default image selected in the instance creation interface. Purging properties removes the 'default' state.
 
== Deleting Unused Images ==
 
{{Warning | '''WARNING!''' You should never delete old images from Glance as long as there are VMs based on those base images.}}
 
Don't delete glance images as long as there are VMs runnning them:
* Resizing a VM requires access to the base image it was launched with. See how [[Portal:Cloud_VPS/Admin/Resizing_instances|resizing instances]] works to understand it.
* We should always be able to identify the OS an instance is running based on the image name.
 
You can locate unused images with the wmcs-imageusage script:
 
<syntaxhighlight lang="bash">
root@cloudcontrol1003:~# wmcs-imageusage
-- unknown image 249c601a-42c8-4ba7-ba7a-878fba4ef799
-- unknown image 68e61634-d599-4c6b-94da-10e6c7d36573
ea564b1c-9e5d-45a9-9d53-2fe023344e3f: debian-9.13-stretch (deprecated 2021-01-22), 0
a64f590c-3ed7-43e1-a592-b44d86f10641: debian-10.0-buster (deprecated 2019-07-30), 0
86105151-0d2e-498d-95b3-da712f19c7e2: debian-9.0-stretch (deprecated 2017-08-03), 0
559facb6-532f-47b0-9fa2-f0c0207c00c9: debian-9.13-stretch, 1
baa92f56-8aca-4855-a0f6-47d94e8a2167: debian-9.11-stretch (deprecated 2019-12-18), 1
6255ff81-db78-4521-b770-076fe16d365f: debian-9.11-stretch (deprecated 2019-12-15), 1
d5d88ba0-0def-4cfb-8c7c-cae8946722d8: debian-8.0-jessie (deprecated 2015-06-13), 1
160313be-b1b1-4b62-8906-bee26f25f6dc: debian-9.13-stretch (deprecated 2021-03-11), 2
f345f03d-a601-48cf-951b-ad1b4ad8f551: debian-9.13-stretch (deprecated 2021-01-20), 2
5cd88504-5e2b-4f63-b18e-2ec935c914dd: debian-10.0-buster-prerelease, 2
b7c4fc02-433f-4b49-97c2-b2492012b742: debian-8.2-jessie (deprecated 2016-02-16), 2
d633c111-00d1-4e72-bb3b-c2bd60518f4a: debian-10.0-buster, 3
fc6fb78b-4515-4dcc-8254-591b9fe01762: debian-10.0-buster (deprecated 2019-12-18), 3
e27bab24-4a17-47e5-bb89-b0c7526dea20: debian-9.2-stretch (deprecated 2017-12-13), 3
b7616101-3b9e-4948-98c7-25582234e788: debian-9.0-stretch (deprecated 2017-09-27), 3
0e33dcf3-37fe-416c-bc00-3abd84dda054: debian-9.0-stretch (deprecated 2017-07-19), 3
ad7bee1a-a890-4fed-b851-6c02138683b0: debian-9.13-stretch (deprecated 2021-03-01), 4
374ca3c6-6c4b-4e03-9f31-52e0d44aae0c: debian-10.0-buster (deprecated 2019-07-29), 4
d620d77c-c023-41ae-944c-2f10063bfc77: debian-9.6-stretch (deprecated 2019-02-14), 4
bb37bd5c-3cc5-4ee6-82f8-473e9568ff44: debian-9.3-stretch (deprecated 2018-01-10), 4
13b0883f-2ce6-467a-ba18-17484413faaa: debian-9.8-stretch (deprecated 2019-04-02), 7
e02770ae-b45f-4776-a852-d9a13217611e: debian-9.1-stretch (deprecated  2017-11-16), 7
325ce1c8-2f95-4e98-8088-f7b46e7a6bb5: debian-9.9-stretch (deprecated 2019-09-21), 8
e3716d55-5278-4f9b-a30f-41db9cb23ef8: debian-9.7-stretch (deprecated 2019-03-14), 10
b7274e93-30f4-4567-88aa-46223c59107e: debian-9.4-stretch (deprecated 2018-08-01), 12
73ccc348-f69d-4cd5-9f09-d83ea222fa02: debian-9.11-stretch (deprecated 2019-11-05), 13
64351116-a53e-4a62-8866-5f0058d89c2b: debian-10.0-buster (deprecated 2021-03-01), 18
6e6d743a-64f0-4cd3-b1b9-327bbf57e03b: debian-9.3-stretch (deprecated 2018-06-05), 19
25eeb420-1977-45f2-bbd9-fb65f48c0947: debian-9.11-stretch (deprecated 2020-10-17), 21
10783e59-b30e-4426-b509-2fbef7d3103c: debian-9.8-stretch (deprecated 2019-07-25), 30
e971dc0f-3b5c-4cd2-ab8b-02faf403c136: debian-10.0-buster (deprecated 2021-03-24), 46
c6273cce-9b8b-4364-9f1f-7bf58436994f: debian-9.5-stretch (deprecated 2018-11-22), 49
b6b58ba2-8656-49b4-af13-d0530ac05365: debian-10.0-buster (deprecated 2019-12-15), 71
7c6371d1-8411-48c7-bf73-2ef6d6ff2a15: debian-9.6-stretch (deprecated 2019-01-22), 83
6b67c8a1-6356-464d-a885-0576d7263e51: debian-10.0-buster (deprecated 2021-02-22), 92
031d2d76-8368-4066-a502-d28107d0195e: debian-10.0-buster (deprecated 2020-10-16), 227
</syntaxhighlight>
 
Images are cheap, though, so generally image cleanup is unwarranted. Typically we only clean up images that were explicitly single-use (e.g. created to troubleshoot the image creation logic) or images for distributions that we no longer support and which no longer exist within our cloud (e.g. Debian Jessie.)
 
Deleting an image from glance is a single command:
 
<syntaxhighlight lang="bash">
root@cloudcontrol1003:~# openstack image delete <imageid>
</syntaxhighlight>


<source lang="bash">
If this fails with a warning that the image is used by VMs, stop now!  Often, though, this command will fail due to out-of-band snapshots (e.g. those created by the image backup service.)
qemu-system-x86_64 ./<new image name>.raw --curses
</source>


Unfortunately, qemu's default behavior is to suppress all boot logs, so you'll be looking at a mostly-blank screen for several minutes before getting a login prompt with no working password.  Turning on a working login account for test purposes is left as an exercise to the reader -- bootstrap-vz creates one by default (login: root passwd:test) but our config wisely disables it.
<syntaxhighlight lang="bash">
root@cloudcontrol1003:~# openstack image delete 11d52ebe-e4a0-45af-99ea-8f286ed55696
Failed to delete image with name or ID '11d52ebe-e4a0-45af-99ea-8f286ed55696': HTTP 409 Conflict: The image cannot be deleted because it has snapshot(s).
Failed to delete 1 of 1 images.
</syntaxhighlight>


Bootstrap-vz uses source files from /etc/bootstrap-vz.  These files are puppetized, so you'll want to disable puppet if you change them.
To delete all snapshots for a given image, use 'rbd snap purge':


Bootstrap-vz also uses several source files that are standard local config files on the build host.  For a complete list of these files, look at the 'file_copy:' section in /etc/bootstrap-vz/manifests/labs-jessie.manifest.yaml
<syntaxhighlight lang="bash">
root@cloudcontrol1003:~# rbd snap purge eqiad1-glance-images/11d52ebe-e4a0-45af-99ea-8f286ed55696
Removing all snapshots: 100% complete...done.
</syntaxhighlight>


The build process for Stretch is similar; the build system is labs-bootstrapvz-stretch.openstack.eqiad.wmflabs and the manifest to use is named labs-stretch.manifest.yaml.
Once snapshots are removed the image can be removed with 'openstack image delete'.


Another method is to run:
== {{anchor|Private Images}} Restricted images ==
It's possible to restrict an image to a single project or set of projects.  This is useful when e.g. gradually deprecating use of an OS.


<source lang="bash">
First, mark the image as "shared" and restrict it to the desired project:
sudo qemu-system-x86_64 -nographic -serial mon:stdio -enable-kvm ./debian-jessie.raw
<syntaxhighlight lang="shell-session">
</source>
$ ssh cloudcontrol1004.wikimedia.org
$ sudo wmcs-openstack image set --property visibility=shared --project $PROJECT_ID $IMAGE_UUID
$ sudo wmcs-openstack image set --activate $IMAGE_UUID
</syntaxhighlight>


This way you will see all the boot prompt.
Then, allow the 'observer' user to access it so that [[toolforge:openstack-browser|OpenStack browser]] can display something more useful than "UNKNOWN" for instances created from it:
<syntaxhighlight lang="shell-session">
$ sudo su -
$ source ~/novaenv.sh
$ glance member-create $IMAGE_UUID observer
$ glance member-update $IMAGE_UUID observer accepted
</syntaxhighlight>
{{Fixme|1=can this be done with <code>openstack</code> instead of <code>glance</code>?}}


== Install a new image ==
== How To Deactive Obsolete Images ==


On cloudcontrol1003 (or other control box):
We used to use the 'show=true' property to ensure that an image appeared on Wikitech. Now instead we use the image state, where only images with state=active appear in the GUI (both on wikitech and in Horizon.) To deactivate your obsolete image:


First, get the new .qcow2 images into /tmp on cloudcontrol1003.wikimedia.org. One way is to rsync it from the builder VM to your laptop using your labs/labs root key, and rsync it to cloudcontrol1003 using your prod key. For example: <syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
rsync -e 'ssh -i ~/.ssh/id_rsa_labs_private' root@vmbuilder-trusty.openstack.eqiad.wmflabs:/srv/vmbuilder/ubuntu-trusty/tmpUUpMh4.qcow2 .
$ sudo wmcs-openstack image set --deactivate <image-id>
rsync -e 'ssh -i ~/.ssh/id_rsa_prod_new' tmpUUpMh4.qcow2 madhuvishy@cloudcontrol1003.wikimedia.org:/tmp/
</syntaxhighlight>
</syntaxhighlight>


Then, also on cloudcontrol1003...
If you need to reactive it for some reason:


<source lang="bash">
<syntaxhighlight lang="bash">
sudo su -
$ sudo wmcs-openstack image set --activate <image-id>
source ~/novaenv.sh
</syntaxhighlight>
cd /tmp
openstack image create --file ubuntu-trusty.qcow2 --disk-format "qcow2" --container-format "ovf" --public "ubuntu-14.04-trusty (testing)"
openstack image create --file debian-jessie.qcow2 --disk-format "qcow2" --container-format "ovf" --public "debian-8.1-jessie (testing)"
# Test the images by booting instances in labs; if they don't work, delete
# the instances, then delete the images (using glance delete), then
# restart the process
glance index # image-list in mitaka
# find image ids
glance image-update --name "ubuntu-14.04-trusty (deprecated <date>)" <old-trusty-id> --purge-props
glance image-update --name "ubuntu-14.04-trusty" <new-trusty-id> --property show=true --property default=true
glance image-update --name "ubuntu-12.04-precise (deprecated <date>)" <old-precise-id> --purge-props
glance image-update --name "ubuntu-12.04-precise" <new-precise-id> --property show=true
</source>


Notice in the above glance image-update commands the use of properties. If default=true the image will be the default image selected in the instance creation interface; purging properties removes the 'default' state.
Please note that we usually just "deprecate" images by changing their names. Deactivating an image is a more extreme step to be used when you do not want any users to have access to it.


We used to use the 'show=true' property to ensure that an image appeared on wikitech.  Now instead we use the image state, where only images with state=active appear in the gui (both on wikitech and in Horizon.)  To deactivate your obsolete image:
== Internals ==


<source lang="bash">
=== bootstrap-vz configuration files ===
source /etc/novaenv.sh
openstack token issue
curl -X POST -H 'X-Auth-Token: <token id>' http://cloudcontrol1003.wikimedia.org:9292/v2/images/<image id>/actions/deactivate
</source>


To reactivate an image (because it was deactivated in error, or in order to permit a migration):
Bootstrap-vz uses source files from /etc/bootstrap-vz. These files are puppetized, so you'll want to disable Puppet if you change them.


<source lang="bash">
{| class="wikitable"
source /etc/novaenv.sh
|-
openstack token issue
! OS !! Configuration File
curl -X POST -H 'X-Auth-Token: <token id>' http://cloudcontrol1003.wikimedia.org:9292/v2/images/<image id>/actions/reactivate
|-
</source>
| Debian Stretch || /etc/bootstrap-vz/manifests/labs-stretch.manifest.yaml
|-
| Debian Jessie || /etc/bootstrap-vz/manifests/labs-jessie.manifest.yaml
|}


== Delete an old image ==


'''TODO:''' fill me.
Bootstrap-vz also uses several source files that are standard local config files on the build host. For a complete list of these files, look at the 'file_copy:' section in /etc/bootstrap-vz/manifests/labs-{stretch,jessie}.manifest.yaml


= First boot =
=== First Boot ===


The first boot of the VM image is a key moment in the setup of an instance in Cloud VPS.
The first boot of the VM image is a key moment in the setup of an instance in Cloud VPS.
Line 185: Line 324:
Until the last point, the instance may have limited connectivity or usability.
Until the last point, the instance may have limited connectivity or usability.


= Common issues and hacks =
== Troubleshooting ==


These problems may vary from deployment to deployment (main, labtest, labtestn), but they could be common.
{{note | For general Cloud VPS troubleshooting, please read [[Portal:Cloud_VPS/Admin/Troubleshooting | the operational troubleshooting documents]].}}
This troubleshooting section is specific for VM images (in glance) and generally only usefull when dealing with new VM images.
 
=== Common Issues ===
 
Common issues when dealing with VM images. These problems may vary from [[Portal:Cloud_VPS/Admin/Deployments | deployment]] to deployment, but they could be common.


* Image does not have the puppet master CA, so it fails to fetch catalog (see [[phab:T181523]])
* Image does not have the puppet master CA, so it fails to fetch catalog (see [[phab:T181523]])
* Image does not have the puppet master CRL, so it fails to fetch catalog (see [[phab:T181523]])
* Image does not have the puppet master CRL, so it fails to fetch catalog (see [[phab:T181523]])
* Image doesn't correctly resolve the hostname/domain name (so it fails to fetch is own puppet catalog)
* Image doesn't correctly resolve the hostname/domain name (so it fails to fetch its own puppet catalog)


== Inspecting a disk by hand ==
=== How To Inspect Disk Contents ===


If you want to explore & edit the disk image of a live instance, follow this procedure:
If you want to explore and edit the disk image of a ''live instance'', read the docs at [[Portal:Cloud_VPS/Admin/Troubleshooting#Mounting_an_instance_disk | Cloud VPS troubleshooting, mounting an instance disk]].


* locate the virt server in which the VM is running
=== How to fix VM disk corruption ===
<syntaxhighlight lang="shell-session>
aborrero@labtestvirt2001:~ $ for i in $(sudo virsh list --all | grep i.* | awk -F' ' '{print $2}') ; do echo -n "$i " ; sudo virsh dumpxml $i | grep nova:name ; done
i-0000025d      <nova:name>puppettestingui</nova:name>
i-00000262      <nova:name>aptproxy2</nova:name>
i-00000263      <nova:name>t2</nova:name>
i-00000264      <nova:name>t3</nova:name>
</syntaxhighlight>


* Once you know the internal instance name (i-xxxxx), locate the disk file
Please read [[Portal:Cloud_VPS/Admin/Troubleshooting#Fix_VM_disk_corruption_(fsck) | Cloud VPS troubleshooting, fixing VM disk corruption (fsck)]].
<syntaxhighlight lang="shell-session>
aborrero@labtestvirt2001:~ $ sudo virsh dumpxml i-00000264 | grep "source file" | grep disk
      <source file='/var/lib/nova/instances/09865310-b440-4dc7-99ab-fb5f35be04fb/disk'/>
</syntaxhighlight>


* shutdown the machine (from inside, from horizon, using virsh or whatever)
== Building an image from an existing VM ==
* copy the disk file to your home
<syntaxhighlight lang="shell-session>
aborrero@labtestvirt2001:~ $ cp /var/lib/nova/instances/09865310-b440-4dc7-99ab-fb5f35be04fb/disk t3-disk.qcow2
</syntaxhighlight>


* disable puppet and install libguestfs-tools
Nova can create a new base image out of an existing VM. This isn't something that we've done much but it's potentially useful to e.g. rapidly create a fleet of worker nodes from an original template.
<syntaxhighlight lang="shell-session>
aborrero@labtestvirt2001:~ $ sudo puppet agent --disable "inspecting instance disk" ; sudo aptitude install libguestfs-tools -y
[...]
</syntaxhighlight>


* create a destination directory and mount the disk!
<syntaxhighlight lang="shell-session>
aborrero@labtestvirt2001:~ $ mkdir mnt ; sudo guestmount -a t3-disk.qcow2 -m /dev/sda3 --rw mnt
</syntaxhighlight>


* you can now read/write the instance disk in the mount point
* Stop the source VM
* when done, umount, copy back the instance disk and start the instance!
openstack server stop <instance-id>
* Create the new image
openstack server image create <instance-id> --name <new image name>
* Wait for the image state to change from 'queued' to 'saving' to 'active'
openstack image show <new image id>
* Set project ownership
openstack image set --shared <new image id>
openstack image add project <new image id> <project>


= See also =
== See Also ==


* [[Portal:Cloud_VPS/Admin/Maintenance]]: information on cloudvps maintenance tasks
* [[Portal:Cloud_VPS/Admin/Maintenance]]: Information on Cloud VPS maintenance tasks


[[Category:VPS admin]]
[[Category:VPS admin]]

Latest revision as of 19:01, 4 September 2021

Cloud VPS uses special VM images that contain all the customizations required for our environment.

Builders

Images for Debian Buster (and subsequent release) are based off of official debian upstream builds. These are built on a cloudcontrol node.

Due to an old version of cloud-init that is bundled with upstream Stretch, we build custom base images (with modern cloud-init) for Stretch. Those are built on with bootstrap-vz on cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs.

Building Pre-puppetized Images with wmcs-image-create

Our cloud should work with any upstream Debian base image that includes cloud-init version later than 18.4. The Debian project provides ready-made images prepared for use with OpenStack.

For faster startup and better storage performance, we typically use a pre-puppetized snapshot based on an upstream image. The wmcs-image-create script will automatically generate and install a pre-puppetized image. This script is invoked with a url pointing to the desired upstream image to use as a starting point. It boots a temporary VM using that image, puppetized that VM, and creates a new image based on that puppetized VM.

root@cloudcontrol1003:~# wmcs-image-create --image-url https://cdimage.debian.org/cdimage/openstack/10.8.3-20210304/debian-10.8.3-20210304-openstack-amd64.raw --new-image-name "debian-10 example build" --project-owner "testlabs"
Downloading upstream image...
--2021-03-25 19:54:41--  https://cdimage.debian.org/cdimage/openstack/10.8.3-20210304/debian-10.8.3-20210304-openstack-amd64.raw
Resolving cdimage.debian.org (cdimage.debian.org)... 2001:6b0:19::165, 2001:6b0:19::173, 194.71.11.165, ...
Connecting to cdimage.debian.org (cdimage.debian.org)|2001:6b0:19::165|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://saimei.ftp.acc.umu.se/cdimage/openstack/10.8.3-20210304/debian-10.8.3-20210304-openstack-amd64.raw [following]
--2021-03-25 19:54:42--  https://saimei.ftp.acc.umu.se/cdimage/openstack/10.8.3-20210304/debian-10.8.3-20210304-openstack-amd64.raw
Resolving saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)... 2001:6b0:19::138, 194.71.11.138
Connecting to saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)|2001:6b0:19::138|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2147483648 (2.0G)
Saving to: ‘/usr/local/sbin/wmcs-image-createikq36pde/upstreamimage’

/usr/local/sbin/wmcs-image-createikq36 100%[===========================================================================>]   2.00G  64.8MB/s    in 33s     

2021-03-25 19:55:16 (61.5 MB/s) - ‘/usr/local/sbin/wmcs-image-createikq36pde/upstreamimage’ saved [2147483648/2147483648]

Loading the upstream image into glance
Launching a vm with the new image
Created temporary VM dc7ad6ed-b7d0-4631-b034-630b66ea0450
Sleeping 5 minutes while we wait for the VM to start up...
Waiting one minutes for VM to puppetize
Stopping the VM
Taking a snapshot of the stopped instance
Creating snapshot ec347068-949f-42c9-9ded-000c165b3b50
Waiting for snapshot to finish saving...
Grabbing handle to snapshot data
Downloading snapshot to /usr/local/sbin/wmcs-image-createikq36pde/snapshot.img
Making snapshot file sparse
Creating final image 15cd26bc-15a5-4b8e-b0bb-cc1bd6c5ca24
Setting image ownership and visibility
Cleaning up intermediate VM
Cleaning up VM snapshot
Cleaning up upstream image
Finished creating new image: 15cd26bc-15a5-4b8e-b0bb-cc1bd6c5ca24

The total run should take 10-15 minutes. Once the new image has been tested it needs to be renamed and marked public so that it's available to all projects:

# note IDs of outgoing and incoming image
root@cloudcontrol1003:~#  openstack image list
# Rename and deactivate the former image
root@cloudcontrol1003:~#  openstack image set --name "debian-10.0-buster (deprecated yyyy-mm-dd)" --deactivate <former active image id>
# Publish the new image
root@cloudcontrol1003:~#  openstack image set --name "debian-10.0-buster (deprecated yyyy-mm-dd)" --public <new image build id>

Legacy Build Process for Debian Stretch

Building with bootstrap-vz

  • Login to cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs
  • Build and convert image
sudo su -
cd /target
bootstrap-vz --pause-on-error  /etc/bootstrap-vz/manifests/labs-stretch.manifest.yaml
qemu-img convert -f raw -O qcow2 debian-stretch-amd64-$(date "+%Y%m%d").raw debian-stretch-amd64-$(date "+%Y%m%d").qcow2

Local bootstrap-vz for stretch

The version of bootstrap-vz built for Stretch doesn't support setting mount options in the manifest. Since we want 'discard' set, the local install of bootstrapvz on cloud-bootstrapvz-stretch.openstack.eqiad1.wikimedia.cloud has a hack in place to make this setting, in bootstrapvz/common/tasks/filesystem.py:

190c190
<             mount_opts = ['defaults']
---
>             mount_opts = ['discard', 'defaults']

How To Test

You can boot an image locally for testing, like this:

sudo qemu-system-x86_64 -nographic -serial mon:stdio -enable-kvm image_name.raw

If the command above does not work, you try can try the following command (beware boot logs will be supressed):

qemu-system-x86_64 image_name.raw --curses

Having a working login account for test purposes is left as an exercise to the reader. bootstrap-vz creates one by default (login:root / passwd:test) but our config wisely disables it.

How To Deploy

Images are deployed to the OpenStack Glance service.

  • Copy the .qcow2 file to /tmp on the cloudcontrol1003.wikimedia.org server

Since the file has to cross the Cloud VPS / Production boundary, you can copy it from the builder server to your laptop (using your Cloud Services root key) and then from your laptop to cloudcontrol1003 (using your production key):

rsync --progress -v -e ssh root@cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs:/target/debian-stretch-amd64-$(date "+%Y%m%d").qcow2 .
rsync --progress -v -e ssh debian-stretch-amd64-$(date "+%Y%m%d").qcow2 cloudcontrol1003.wikimedia.org:/tmp/

Alternatively, you can open a temporary HTTP server to make this transfer:

cloud-bootstrapvz-stretch:~ $ cd /target
cloud-bootstrapvz-stretch:~ $ python3 -m http.server 80

cloudcontrol1003:/tmp$ wget http://185.15.56.45/debian-stretch-amd64-$(date "+%Y%m%d").qcow2
  • Login to cloudcontrol1003.wikimedia.org
  • Convert the image from qcow2 to raw/sparse. This is a new step necessary now that images are stored using ceph/rbd:
qemu-img convert -f qcow2 -O raw  ./<filename>.qcow2 ./<filename>.raw.notsparse
cp --sparse=always ./<filename>.raw.notsparse  ./<filename>.raw
  • Create new image in Glance:
sudo su -
source ~/novaenv.sh 
cd /tmp
openstack image create --file debian-stretch-amd64-$(date "+%Y%m%d").raw --disk-format "raw" --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi --container-format "ovf" --public "debian-9.6-stretch"
  • Test new image by booting a new VM with it (if the image is faulty, remember to delete the test VM and the faulty image)
  • Update fullstackd to use this new image (see T218314 for an example).
  • Get a list of existing images
openstack image list
  • Append "deprecated" to old images and remove properties (only if new image is working as expected)
openstack image set --name "debian-9.5-stretch (deprecated <date>)" <old-image-id>
nova image-meta <old-image-id> delete default
nova image-meta <old-image-id> delete show

Passing --purge-props to openstack image set should be enough to clear all properties but it's currently not available in our OpenStack version. The nova image-meta commands serve the same purpose but you have to delete each property individually. This should be reviewed when OpenStack is upgraded.

  • Make the new image the default for new instances
openstack image set --name "debian-9.6-stretch" <new-image-id> --property show=true --property default=true

Notice in the above glance image-update commands the use of properties. If default=true the image will be the default image selected in the instance creation interface. Purging properties removes the 'default' state.

Deleting Unused Images

Don't delete glance images as long as there are VMs runnning them:

  • Resizing a VM requires access to the base image it was launched with. See how resizing instances works to understand it.
  • We should always be able to identify the OS an instance is running based on the image name.

You can locate unused images with the wmcs-imageusage script:

root@cloudcontrol1003:~# wmcs-imageusage 
 -- unknown image 249c601a-42c8-4ba7-ba7a-878fba4ef799
 -- unknown image 68e61634-d599-4c6b-94da-10e6c7d36573
ea564b1c-9e5d-45a9-9d53-2fe023344e3f: debian-9.13-stretch (deprecated 2021-01-22), 0
a64f590c-3ed7-43e1-a592-b44d86f10641: debian-10.0-buster (deprecated 2019-07-30), 0
86105151-0d2e-498d-95b3-da712f19c7e2: debian-9.0-stretch (deprecated 2017-08-03), 0
559facb6-532f-47b0-9fa2-f0c0207c00c9: debian-9.13-stretch, 1
baa92f56-8aca-4855-a0f6-47d94e8a2167: debian-9.11-stretch (deprecated 2019-12-18), 1
6255ff81-db78-4521-b770-076fe16d365f: debian-9.11-stretch (deprecated 2019-12-15), 1
d5d88ba0-0def-4cfb-8c7c-cae8946722d8: debian-8.0-jessie (deprecated 2015-06-13), 1
160313be-b1b1-4b62-8906-bee26f25f6dc: debian-9.13-stretch (deprecated 2021-03-11), 2
f345f03d-a601-48cf-951b-ad1b4ad8f551: debian-9.13-stretch (deprecated 2021-01-20), 2
5cd88504-5e2b-4f63-b18e-2ec935c914dd: debian-10.0-buster-prerelease, 2
b7c4fc02-433f-4b49-97c2-b2492012b742: debian-8.2-jessie (deprecated 2016-02-16), 2
d633c111-00d1-4e72-bb3b-c2bd60518f4a: debian-10.0-buster, 3
fc6fb78b-4515-4dcc-8254-591b9fe01762: debian-10.0-buster (deprecated 2019-12-18), 3
e27bab24-4a17-47e5-bb89-b0c7526dea20: debian-9.2-stretch (deprecated 2017-12-13), 3
b7616101-3b9e-4948-98c7-25582234e788: debian-9.0-stretch (deprecated 2017-09-27), 3
0e33dcf3-37fe-416c-bc00-3abd84dda054: debian-9.0-stretch (deprecated 2017-07-19), 3
ad7bee1a-a890-4fed-b851-6c02138683b0: debian-9.13-stretch (deprecated 2021-03-01), 4
374ca3c6-6c4b-4e03-9f31-52e0d44aae0c: debian-10.0-buster (deprecated 2019-07-29), 4
d620d77c-c023-41ae-944c-2f10063bfc77: debian-9.6-stretch (deprecated 2019-02-14), 4
bb37bd5c-3cc5-4ee6-82f8-473e9568ff44: debian-9.3-stretch (deprecated 2018-01-10), 4
13b0883f-2ce6-467a-ba18-17484413faaa: debian-9.8-stretch (deprecated 2019-04-02), 7
e02770ae-b45f-4776-a852-d9a13217611e: debian-9.1-stretch (deprecated  2017-11-16), 7
325ce1c8-2f95-4e98-8088-f7b46e7a6bb5: debian-9.9-stretch (deprecated 2019-09-21), 8
e3716d55-5278-4f9b-a30f-41db9cb23ef8: debian-9.7-stretch (deprecated 2019-03-14), 10
b7274e93-30f4-4567-88aa-46223c59107e: debian-9.4-stretch (deprecated 2018-08-01), 12
73ccc348-f69d-4cd5-9f09-d83ea222fa02: debian-9.11-stretch (deprecated 2019-11-05), 13
64351116-a53e-4a62-8866-5f0058d89c2b: debian-10.0-buster (deprecated 2021-03-01), 18
6e6d743a-64f0-4cd3-b1b9-327bbf57e03b: debian-9.3-stretch (deprecated 2018-06-05), 19
25eeb420-1977-45f2-bbd9-fb65f48c0947: debian-9.11-stretch (deprecated 2020-10-17), 21
10783e59-b30e-4426-b509-2fbef7d3103c: debian-9.8-stretch (deprecated 2019-07-25), 30
e971dc0f-3b5c-4cd2-ab8b-02faf403c136: debian-10.0-buster (deprecated 2021-03-24), 46
c6273cce-9b8b-4364-9f1f-7bf58436994f: debian-9.5-stretch (deprecated 2018-11-22), 49
b6b58ba2-8656-49b4-af13-d0530ac05365: debian-10.0-buster (deprecated 2019-12-15), 71
7c6371d1-8411-48c7-bf73-2ef6d6ff2a15: debian-9.6-stretch (deprecated 2019-01-22), 83
6b67c8a1-6356-464d-a885-0576d7263e51: debian-10.0-buster (deprecated 2021-02-22), 92
031d2d76-8368-4066-a502-d28107d0195e: debian-10.0-buster (deprecated 2020-10-16), 227

Images are cheap, though, so generally image cleanup is unwarranted. Typically we only clean up images that were explicitly single-use (e.g. created to troubleshoot the image creation logic) or images for distributions that we no longer support and which no longer exist within our cloud (e.g. Debian Jessie.)

Deleting an image from glance is a single command:

root@cloudcontrol1003:~# openstack image delete <imageid>

If this fails with a warning that the image is used by VMs, stop now! Often, though, this command will fail due to out-of-band snapshots (e.g. those created by the image backup service.)

root@cloudcontrol1003:~# openstack image delete 11d52ebe-e4a0-45af-99ea-8f286ed55696
Failed to delete image with name or ID '11d52ebe-e4a0-45af-99ea-8f286ed55696': HTTP 409 Conflict: The image cannot be deleted because it has snapshot(s).
Failed to delete 1 of 1 images.

To delete all snapshots for a given image, use 'rbd snap purge':

root@cloudcontrol1003:~# rbd snap purge eqiad1-glance-images/11d52ebe-e4a0-45af-99ea-8f286ed55696
Removing all snapshots: 100% complete...done.

Once snapshots are removed the image can be removed with 'openstack image delete'.

Restricted images

It's possible to restrict an image to a single project or set of projects. This is useful when e.g. gradually deprecating use of an OS.

First, mark the image as "shared" and restrict it to the desired project:

$ ssh cloudcontrol1004.wikimedia.org
$ sudo wmcs-openstack image set --property visibility=shared --project $PROJECT_ID $IMAGE_UUID
$ sudo wmcs-openstack image set --activate $IMAGE_UUID

Then, allow the 'observer' user to access it so that OpenStack browser can display something more useful than "UNKNOWN" for instances created from it:

$ sudo su -
$ source ~/novaenv.sh
$ glance member-create $IMAGE_UUID observer
$ glance member-update $IMAGE_UUID observer accepted

How To Deactive Obsolete Images

We used to use the 'show=true' property to ensure that an image appeared on Wikitech. Now instead we use the image state, where only images with state=active appear in the GUI (both on wikitech and in Horizon.) To deactivate your obsolete image:

$ sudo wmcs-openstack image set --deactivate <image-id>

If you need to reactive it for some reason:

$ sudo wmcs-openstack image set --activate <image-id>

Please note that we usually just "deprecate" images by changing their names. Deactivating an image is a more extreme step to be used when you do not want any users to have access to it.

Internals

bootstrap-vz configuration files

Bootstrap-vz uses source files from /etc/bootstrap-vz. These files are puppetized, so you'll want to disable Puppet if you change them.

OS Configuration File
Debian Stretch /etc/bootstrap-vz/manifests/labs-stretch.manifest.yaml
Debian Jessie /etc/bootstrap-vz/manifests/labs-jessie.manifest.yaml


Bootstrap-vz also uses several source files that are standard local config files on the build host. For a complete list of these files, look at the 'file_copy:' section in /etc/bootstrap-vz/manifests/labs-{stretch,jessie}.manifest.yaml

First Boot

The first boot of the VM image is a key moment in the setup of an instance in Cloud VPS.

This is usually done by means of the /root/firstboot.sh script which is called by means of /etc/rc.local.

The script will do:

  • some LVM configuration
  • run DHCP request for configuration
  • name/domain resolution to autoconfigure the VM
  • initial puppet autoconfiguration (cert request, etc)
  • initial configuration of nscd/nslcd
  • initial apt updates
  • NFS mounts if required
  • final puppet run to fetch all remaining configuration (ssh keys, packages, etc)

Until the last point, the instance may have limited connectivity or usability.

Troubleshooting

This troubleshooting section is specific for VM images (in glance) and generally only usefull when dealing with new VM images.

Common Issues

Common issues when dealing with VM images. These problems may vary from deployment to deployment, but they could be common.

  • Image does not have the puppet master CA, so it fails to fetch catalog (see phab:T181523)
  • Image does not have the puppet master CRL, so it fails to fetch catalog (see phab:T181523)
  • Image doesn't correctly resolve the hostname/domain name (so it fails to fetch its own puppet catalog)

How To Inspect Disk Contents

If you want to explore and edit the disk image of a live instance, read the docs at Cloud VPS troubleshooting, mounting an instance disk.

How to fix VM disk corruption

Please read Cloud VPS troubleshooting, fixing VM disk corruption (fsck).

Building an image from an existing VM

Nova can create a new base image out of an existing VM. This isn't something that we've done much but it's potentially useful to e.g. rapidly create a fleet of worker nodes from an original template.


  • Stop the source VM
openstack server stop <instance-id>
  • Create the new image
openstack server image create <instance-id> --name <new image name>
  • Wait for the image state to change from 'queued' to 'saving' to 'active'
openstack image show <new image id>
  • Set project ownership
openstack image set --shared <new image id>
openstack image add project <new image id> <project>

See Also