You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Cowbuilder: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Alex Monk
(→‎Stock Debian Jessie: fix unclosed tt causing style issues)
imported>Alexandros Kosiaris
(Merge [cowbuilder on labs] into this page)
Line 1: Line 1:
''This is a generic guide for cowbuilder, to use it on wmflabs see [[Cowbuilder on labs]]''
''This page is now a pretty simple stub. Feel free to look at the history of it but almost everything should be automagically managed by puppet if you do the following.''


''Apply role::package::builder on a labs VM will allow you to avoid all the following as it is already being done by puppet. Documentation is kept up to date at:''
Apply role::package::builder on a labs VM Documentation is kept up to date at
https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/
https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/


== Intro ==
= Labs =
cowbuilder is a Debian packaging tool that takes advantages of Copy On Write (COW) to speed up the building process. You first bootstrap Debian distributions you are interested in and configure them as it may fit your use case. When building a package, cowbuilder hardlink the files in a new hierarchy and would write to the disk only on actual change to the files. That makes the startup lightning fast.
Apply <tt>role::labs::lvm::srv</tt> to get additional disk space at /srv.


This page is a step by step guide to setup a cowbuilder environement. It assumes your machine is running Debian/Jessie machine and you have root access. We will cover the initial image creations for stock Debian/Jessie and the various custom distribution Wikimedia is using. We will then look at how they work out when using git-buildpackage.
Apply <tt>role::package::builder</tt>


= cowbuilder images =
[[Category:Package management]]
 
== Installation ==
 
apt-get install build-essential git-buildpackage cowbuilder
 
Done.
 
== Create images ==
 
TODO Configure pbuilder:
 
/root/.pbuilderrc
Is APTCACHE really needed ?
 
== Stock Debian Jessie ==
 
cowbuilder --create --distribution=jessie --debootstrapopts --variant=buildd
 
It takes the Debian mirror from your sources.list and install it in a chroot under <tt>/var/cache/pbuilder/base-jessie.cow</tt>.  That will be known as the distribution <tt>jessie</tt>.  More options are available via <tt>man cowbuilder</tt> and the underlying command being used <tt>man pbuilder</tt>.
 
== Debian Jessie Wikimedia ==
 
Wikimedia uses its own apt repositories which comes with additional packages and backports. You will want to create another jessie flavor image. It would be similar but points to wikimedia and ensure the packages receive an higher priority. For a start, just copy:
 
cp -a /var/cache/pbuilder/base-jessie.cow /var/cache/pbuilder/base-jessie-wikimedia.cow
 
We will further tweak that image by login into it and having our modifications saved back to disk:
 
cowbuilder --login --basepath /var/cache/pbuilder/base-jessie-wikimedia.cow --save-after-login
 
You now have a prompt in this image, add in the Wikimedia repository pinned to receive priority:
 
echo deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty \
  > /etc/apt/sources.list.d/wikimedia.list
echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" \
  > /etc/apt/preferences.d/wikimedia
 
Since our packages are signed with GPG, you will want to grab the gpg keys we use and verify them. To do so:
 
apt-get install wget
wget http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg
apt-key add wikimedia-archive-keyring.gpg
apt-get update
 
You will want to clear out some of the modifications you just made:
 
dpkg --purge wget
apt-get autoremove --purge
rm wikimedia-archive-keyring.gpg
 
And upgrade the image:
 
apt-get upgrade
 
exit / Ctrl+d and cowbuilder will save the result of your modification to the base-jessie-wikimedia.cow image.
 
== Ubuntu images ==
 
TODO
 
    apt-get install ubuntu-archive-keyring
    # We need universe!
    cowbuilder --create \
        --basepath /packaging/base-trusty.cow \
        --mirror http://nova.clouds.archive.ubuntu.com/ubuntu/ \
        --distribution trusty \
        --components 'main universe' \
        --debootstrapopts --variant=buildd \
        --debootstrapopts --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg \
 
  echo 'deb http://apt.wikimedia.org/wikimedia precise-wikimedia main universe thirdparty' > /etc/apt/sources.list.d/wikimedia.list
  echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" > /etc/apt/preferences.d/wikimedia
 
Probably need to get trusty updates as well.
 
=== Result ===
 
An example result of the above steps:
 
<pre>
$ ls -1d /var/cache/pbuilder/*cow
base-jessie.cow
base-jessie-wikimedia.cow
base-precise.cow
base-precise-wikimedia.cow
base-trusty.cow
base-trusty-wikimedia.cow
$
</pre>
 
This let you build a package against either the stock or the Wikimedia flavors for all distributions in use.
 
 
= git-buildpackage =
 
= Maintaining patch series =
 
Placeholder for <tt>gbp pq</tt>.

Revision as of 12:38, 31 January 2016

This page is now a pretty simple stub. Feel free to look at the history of it but almost everything should be automagically managed by puppet if you do the following.

Apply role::package::builder on a labs VM Documentation is kept up to date at https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/

Labs

Apply role::labs::lvm::srv to get additional disk space at /srv.

Apply role::package::builder