You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Cowbuilder: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Alex Monk
(→‎Stock Debian Jessie: fix unclosed tt causing style issues)
imported>Majavah
m (Reverted edits by Fragguni9989 (talk) to last revision by Jobo)
(4 intermediate revisions by 3 users not shown)
Line 1: Line 1:
''This is a generic guide for cowbuilder, to use it on wmflabs see [[Cowbuilder on labs]]''
Documentation available in: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/


''Apply role::package::builder on a labs VM will allow you to avoid all the following as it is already being done by puppet. Documentation is kept up to date at:''
== Introduction ==
https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/
A module used to populate a Debian/Ubuntu package building environment. Meant to be used in the Wikimedia environment but could be adapted for other environments as well.


== Intro ==
== Setting it up ==
cowbuilder is a Debian packaging tool that takes advantages of Copy On Write (COW) to speed up the building process. You first bootstrap Debian distributions you are interested in and configure them as it may fit your use case. When building a package, cowbuilder hardlink the files in a new hierarchy and would write to the disk only on actual change to the files. That makes the startup lightning fast.
Include the package\_builder class in your machine. That can be done with whatever ENC you have puppet working with but in Wikimedia Labs you should create a puppet group, then add the class in the puppet group and just configure your VM with that class


This page is a step by step guide to setup a cowbuilder environement. It assumes your machine is running Debian/Jessie machine and you have root access. We will cover the initial image creations for stock Debian/Jessie and the various custom distribution Wikimedia is using. We will then look at how they work out when using git-buildpackage.
After puppet is done you will have a number of pristine cowbuilder environments. Those would be:


= cowbuilder images =
* stretch-amd64
* buster-amd64
* bullseye-amd64
* sid-amd64


== Installation ==
== Building packages ==
You just either download a ready package:
dget <nowiki>http://http.debian.net/debian/pool/main/d/dh-virtualenv/dh-virtualenv_0.10-1.dsc</nowiki>
export DIST=stretch
sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc
or if you are developing a package and are in the package directory:
DIST=stretch pdebuild
Feel free to change stretch for the distribution of your choice from the ones above


apt-get install build-essential git-buildpackage cowbuilder
== Architectures ==
ARCH=amd64, or ARCH=i386 is supported if you feel like building for different architecture versions. There is no support for other architectures


Done.
== Debugging ==
By default, if the build fails, a hook is executed, providing the user with a shell allowing them to debug the build further. If that's not desired, there's a variable that can be defined to avoid that behavior. Example:
SHELL_ON_FAILURE=no pdebuild
If you reach the conclusion that your build fails because of some effort to write to HOME, and fixing the software to not do that is unfeasible, then you can set
BUILD\_HOME = /build
in .pbuilderrc or /etc/pbuilderrc


== Create images ==
== Using built packages as dependencies ==
By default, cowbuilder will always prefer upstream packages to packages you have built. If you have built a new version of a dependency, you typically want to use that version rather than the one provided by upstream. To force cowbuilder to use packages in the result directory, use:
APT_USE_BUILT=yes sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc


TODO Configure pbuilder:
== Wikimedia repos ==
Aside from sid, the rest of the distributions allow for satisfying build time dependencies via the Wikimedia repos.


/root/.pbuilderrc
To use packages from the Wikimedia repos to satisfy build dependencies during building you can use WIKIMEDIA=yes. There is also the approach of appending -wikimedia to the DIST variable and pbuilder will do what you want.
Is APTCACHE really needed ?


== Stock Debian Jessie ==
Examples:
DIST=stretch-wikimedia pdebuild
WIKIMEDIA=yes DIST=stretch pdebuild
The commands above are equivalent and will both build a package for the stretch distribution using the Wikimedia apt repository.


  cowbuilder --create --distribution=jessie --debootstrapopts --variant=buildd
== Backports repos ==
Packages from the Debian backports repositories can be used to satisfy dependencies as well. To use the backports repository for the distribution selected (e.g. stretch-backports), use either of:
DIST=stretch BACKPORTS=yes pdebuild
  DIST=stretch-backports pdebuild


It takes the Debian mirror from your sources.list and install it in a chroot under <tt>/var/cache/pbuilder/base-jessie.cow</tt>. That will be known as the distribution <tt>jessie</tt>More options are available via <tt>man cowbuilder</tt> and the underlying command being used <tt>man pbuilder</tt>.
== Archived suites ==
Backports is only available for the regular lifetime of a Debian release, not during the LTS stage. After that packages should ideally be rebuilt using internally built components. If this isn't possible for some reason, the ARCHIVE\_BACKPORTS hook can be used to pull in the backports repository from archive.debian.org.
  DIST=buster ARCHIVE\_BACKPORTS=yes pdebuild


== Debian Jessie Wikimedia ==
== Combining Wikimedia and Backports repos ==
Set both WIKIMEDIA and BACKPORTS:
DIST=stretch BACKPORTS=yes WIKIMEDIA=yes pdebuild
When using a distribution suffix, the other repo must be enabled via an environment variable. The following examples are equivalent:
DIST=stretch-backports WIKIMEDIA=yes pdebuild
DIST=stretch-wikimedia BACKPORTS=yes pdebuild


Wikimedia uses its own apt repositories which comes with additional packages and backports. You will want to create another jessie flavor image. It would be similar but points to wikimedia and ensure the packages receive an higher priority. For a start, just copy:
== git-buildpackage ==
git-pbuilder can be used by git-buildpackage to leverage all of the above but instead of DIST and ARCH you need to use --git-dist=$DIST and --git-arch=$ARCH.
WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=stretch
-sa is being used to enforce the original tarball to be included in the .changes file which is a requirement for Wikimedia reprepro.


cp -a /var/cache/pbuilder/base-jessie.cow /var/cache/pbuilder/base-jessie-wikimedia.cow
The --git-no-pbuilder-autoconf/GIT\_PBUILDER\_AUTOCONF=no tells git-pbuilder to forego all attempts to discover the base path, tarball, or configuration file to set up the pbuilder options but rather instead rely on the settings in .pbuilderrc


We will further tweak that image by login into it and having our modifications saved back to disk:
You can make it a default by editing your ~/.gbp.conf:
[buildpackage]
pbuilder = True
pbuilder-autoconf = False
dist = buster
Without "dist = buster" gbp will build for sid by default.


  cowbuilder --login --basepath /var/cache/pbuilder/base-jessie-wikimedia.cow --save-after-login
== Results ==
The resulting deb files should be in /var/cache/pbuilder/result/${DIST}-${ARCH} like:
  /var/cache/pbuilder/result/stretch-amd64/


You now have a prompt in this image, add in the Wikimedia repository pinned to receive priority:
== Notes ==
If you are getting confused over the naming of pbuilder/cowbuilder, here's some info to help you. pbuilder is the actual base software, cowbuilder is an extension to allow pbuilder to use COW (copy on write) instead of slow .tar.gz base files. For all intents and purposes this should be transparent to you as cowbuilder is the default pbuilder builder.


echo deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty \
== Networking ==
  > /etc/apt/sources.list.d/wikimedia.list
cowbuilder/pbuilder block networking using Linux namespaces. Technically speaking an *unshare* is done in those environments, but the effect is that you can expect networking to not work.
echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" \
  > /etc/apt/preferences.d/wikimedia


Since our packages are signed with GPG, you will want to grab the gpg keys we use and verify them. To do so:
If your package requires internet access to build successfully, it will not work.


apt-get install wget
First, try to fix the package. Packages in general should not require internet access to be built for a variety of reasons which will not be explained here.
wget http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg
apt-key add wikimedia-archive-keyring.gpg
apt-get update


You will want to clear out some of the modifications you just made:
If that is impossible/undesirable, then set
USENETWORK=yes
in /etc/pbuilderrc or ~/.pbuilderrc can be used to override that behaviour. Make sure that the building host has internet access though, or else your change will not be useful


dpkg --purge wget
== Cloud VPS ==
apt-get autoremove --purge
Apply <tt>role::labs::lvm::srv</tt> to get additional disk space at /srv.
rm wikimedia-archive-keyring.gpg
Apply <tt>role::package::builder</tt>


And upgrade the image:
[[Category:Package management]]
 
[[Category:SRE Infrastructure Foundations]]
apt-get upgrade
 
exit / Ctrl+d and cowbuilder will save the result of your modification to the base-jessie-wikimedia.cow image.
 
== Ubuntu images ==
 
TODO
 
    apt-get install ubuntu-archive-keyring
    # We need universe!
    cowbuilder --create \
        --basepath /packaging/base-trusty.cow \
        --mirror http://nova.clouds.archive.ubuntu.com/ubuntu/ \
        --distribution trusty \
        --components 'main universe' \
        --debootstrapopts --variant=buildd \
        --debootstrapopts --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg \
 
  echo 'deb http://apt.wikimedia.org/wikimedia precise-wikimedia main universe thirdparty' > /etc/apt/sources.list.d/wikimedia.list
  echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" > /etc/apt/preferences.d/wikimedia
 
Probably need to get trusty updates as well.
 
=== Result ===
 
An example result of the above steps:
 
<pre>
$ ls -1d /var/cache/pbuilder/*cow
base-jessie.cow
base-jessie-wikimedia.cow
base-precise.cow
base-precise-wikimedia.cow
base-trusty.cow
base-trusty-wikimedia.cow
$
</pre>
 
This let you build a package against either the stock or the Wikimedia flavors for all distributions in use.
 
 
= git-buildpackage =
 
= Maintaining patch series =
 
Placeholder for <tt>gbp pq</tt>.

Revision as of 12:28, 26 February 2022

Documentation available in: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/

Introduction

A module used to populate a Debian/Ubuntu package building environment. Meant to be used in the Wikimedia environment but could be adapted for other environments as well.

Setting it up

Include the package\_builder class in your machine. That can be done with whatever ENC you have puppet working with but in Wikimedia Labs you should create a puppet group, then add the class in the puppet group and just configure your VM with that class

After puppet is done you will have a number of pristine cowbuilder environments. Those would be:

  • stretch-amd64
  • buster-amd64
  • bullseye-amd64
  • sid-amd64

Building packages

You just either download a ready package:

dget http://http.debian.net/debian/pool/main/d/dh-virtualenv/dh-virtualenv_0.10-1.dsc
export DIST=stretch
sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc

or if you are developing a package and are in the package directory:

DIST=stretch pdebuild

Feel free to change stretch for the distribution of your choice from the ones above

Architectures

ARCH=amd64, or ARCH=i386 is supported if you feel like building for different architecture versions. There is no support for other architectures

Debugging

By default, if the build fails, a hook is executed, providing the user with a shell allowing them to debug the build further. If that's not desired, there's a variable that can be defined to avoid that behavior. Example:

SHELL_ON_FAILURE=no pdebuild

If you reach the conclusion that your build fails because of some effort to write to HOME, and fixing the software to not do that is unfeasible, then you can set

BUILD\_HOME = /build

in .pbuilderrc or /etc/pbuilderrc

Using built packages as dependencies

By default, cowbuilder will always prefer upstream packages to packages you have built. If you have built a new version of a dependency, you typically want to use that version rather than the one provided by upstream. To force cowbuilder to use packages in the result directory, use:

APT_USE_BUILT=yes sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc

Wikimedia repos

Aside from sid, the rest of the distributions allow for satisfying build time dependencies via the Wikimedia repos.

To use packages from the Wikimedia repos to satisfy build dependencies during building you can use WIKIMEDIA=yes. There is also the approach of appending -wikimedia to the DIST variable and pbuilder will do what you want.

Examples:

DIST=stretch-wikimedia pdebuild
WIKIMEDIA=yes DIST=stretch pdebuild

The commands above are equivalent and will both build a package for the stretch distribution using the Wikimedia apt repository.

Backports repos

Packages from the Debian backports repositories can be used to satisfy dependencies as well. To use the backports repository for the distribution selected (e.g. stretch-backports), use either of:

DIST=stretch BACKPORTS=yes pdebuild
DIST=stretch-backports pdebuild

Archived suites

Backports is only available for the regular lifetime of a Debian release, not during the LTS stage. After that packages should ideally be rebuilt using internally built components. If this isn't possible for some reason, the ARCHIVE\_BACKPORTS hook can be used to pull in the backports repository from archive.debian.org.

DIST=buster ARCHIVE\_BACKPORTS=yes pdebuild

Combining Wikimedia and Backports repos

Set both WIKIMEDIA and BACKPORTS:

DIST=stretch BACKPORTS=yes WIKIMEDIA=yes pdebuild

When using a distribution suffix, the other repo must be enabled via an environment variable. The following examples are equivalent:

DIST=stretch-backports WIKIMEDIA=yes pdebuild
DIST=stretch-wikimedia BACKPORTS=yes pdebuild

git-buildpackage

git-pbuilder can be used by git-buildpackage to leverage all of the above but instead of DIST and ARCH you need to use --git-dist=$DIST and --git-arch=$ARCH.

WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=stretch

-sa is being used to enforce the original tarball to be included in the .changes file which is a requirement for Wikimedia reprepro.

The --git-no-pbuilder-autoconf/GIT\_PBUILDER\_AUTOCONF=no tells git-pbuilder to forego all attempts to discover the base path, tarball, or configuration file to set up the pbuilder options but rather instead rely on the settings in .pbuilderrc

You can make it a default by editing your ~/.gbp.conf:

[buildpackage]
pbuilder = True
pbuilder-autoconf = False
dist = buster

Without "dist = buster" gbp will build for sid by default.

Results

The resulting deb files should be in /var/cache/pbuilder/result/${DIST}-${ARCH} like:

/var/cache/pbuilder/result/stretch-amd64/

Notes

If you are getting confused over the naming of pbuilder/cowbuilder, here's some info to help you. pbuilder is the actual base software, cowbuilder is an extension to allow pbuilder to use COW (copy on write) instead of slow .tar.gz base files. For all intents and purposes this should be transparent to you as cowbuilder is the default pbuilder builder.

Networking

cowbuilder/pbuilder block networking using Linux namespaces. Technically speaking an *unshare* is done in those environments, but the effect is that you can expect networking to not work.

If your package requires internet access to build successfully, it will not work.

First, try to fix the package. Packages in general should not require internet access to be built for a variety of reasons which will not be explained here.

If that is impossible/undesirable, then set

USENETWORK=yes

in /etc/pbuilderrc or ~/.pbuilderrc can be used to override that behaviour. Make sure that the building host has internet access though, or else your change will not be useful

Cloud VPS

Apply role::labs::lvm::srv to get additional disk space at /srv. Apply role::package::builder