You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
This is a generic guide for cowbuilder, to use it on wmflabs see Cowbuilder on labs
Apply role::package::builder on a labs VM will allow you to avoid all the following as it is already being done by puppet. Documentation is kept up to date at: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/
cowbuilder is a Debian packaging tool that takes advantages of Copy On Write (COW) to speed up the building process. You first bootstrap Debian distributions you are interested in and configure them as it may fit your use case. When building a package, cowbuilder hardlink the files in a new hierarchy and would write to the disk only on actual change to the files. That makes the startup lightning fast.
This page is a step by step guide to setup a cowbuilder environement. It assumes your machine is running Debian/Jessie machine and you have root access. We will cover the initial image creations for stock Debian/Jessie and the various custom distribution Wikimedia is using. We will then look at how they work out when using git-buildpackage.
apt-get install build-essential git-buildpackage cowbuilder
TODO Configure pbuilder:
/root/.pbuilderrc Is APTCACHE really needed ?
Stock Debian Jessie
cowbuilder --create --distribution=jessie --debootstrapopts --variant=buildd
It takes the Debian mirror from your sources.list and install it in a chroot under /var/cache/pbuilder/base-jessie.cow. That will be known as the distribution jessie. More options are available via man cowbuilder and the underlying command being used man pbuilder.
Debian Jessie Wikimedia
Wikimedia uses its own apt repositories which comes with additional packages and backports. You will want to create another jessie flavor image. It would be similar but points to wikimedia and ensure the packages receive an higher priority. For a start, just copy:
cp -a /var/cache/pbuilder/base-jessie.cow /var/cache/pbuilder/base-jessie-wikimedia.cow
We will further tweak that image by login into it and having our modifications saved back to disk:
cowbuilder --login --basepath /var/cache/pbuilder/base-jessie-wikimedia.cow --save-after-login
You now have a prompt in this image, add in the Wikimedia repository pinned to receive priority:
echo deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty \ > /etc/apt/sources.list.d/wikimedia.list echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" \ > /etc/apt/preferences.d/wikimedia
Since our packages are signed with GPG, you will want to grab the gpg keys we use and verify them. To do so:
apt-get install wget wget http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg apt-key add wikimedia-archive-keyring.gpg apt-get update
You will want to clear out some of the modifications you just made:
dpkg --purge wget apt-get autoremove --purge rm wikimedia-archive-keyring.gpg
And upgrade the image:
exit / Ctrl+d and cowbuilder will save the result of your modification to the base-jessie-wikimedia.cow image.
apt-get install ubuntu-archive-keyring # We need universe! cowbuilder --create \ --basepath /packaging/base-trusty.cow \ --mirror http://nova.clouds.archive.ubuntu.com/ubuntu/ \ --distribution trusty \ --components 'main universe' \ --debootstrapopts --variant=buildd \ --debootstrapopts --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg \
echo 'deb http://apt.wikimedia.org/wikimedia precise-wikimedia main universe thirdparty' > /etc/apt/sources.list.d/wikimedia.list echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" > /etc/apt/preferences.d/wikimedia
Probably need to get trusty updates as well.
An example result of the above steps:
$ ls -1d /var/cache/pbuilder/*cow base-jessie.cow base-jessie-wikimedia.cow base-precise.cow base-precise-wikimedia.cow base-trusty.cow base-trusty-wikimedia.cow $
This let you build a package against either the stock or the Wikimedia flavors for all distributions in use.
Maintaining patch series
Placeholder for gbp pq.