You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Cowbuilder: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Alex Monk
(→‎Stock Debian Jessie: fix unclosed tt causing style issues)
imported>Jbond
 
(6 intermediate revisions by 5 users not shown)
Line 1: Line 1:
''This is a generic guide for cowbuilder, to use it on wmflabs see [[Cowbuilder on labs]]''
Documentation available in: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/


''Apply role::package::builder on a labs VM will allow you to avoid all the following as it is already being done by puppet. Documentation is kept up to date at:''
== Introduction ==
https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/
A module used to populate a Debian/Ubuntu package building environment. Meant to be used in the Wikimedia environment but could be adapted for other environments as well.


== Intro ==
== Setting it up ==
cowbuilder is a Debian packaging tool that takes advantages of Copy On Write (COW) to speed up the building process. You first bootstrap Debian distributions you are interested in and configure them as it may fit your use case. When building a package, cowbuilder hardlink the files in a new hierarchy and would write to the disk only on actual change to the files. That makes the startup lightning fast.
Include the package\_builder class in your machine. That can be done with whatever ENC you have puppet working with but in Wikimedia Labs you should create a puppet group, then add the class in the puppet group and just configure your VM with that class


This page is a step by step guide to setup a cowbuilder environement. It assumes your machine is running Debian/Jessie machine and you have root access. We will cover the initial image creations for stock Debian/Jessie and the various custom distribution Wikimedia is using. We will then look at how they work out when using git-buildpackage.
After puppet is done you will have a number of pristine cowbuilder environments. Those would be:


= cowbuilder images =
* stretch-amd64
* buster-amd64
* bullseye-amd64
* sid-amd64


== Installation ==
== Building packages ==
You just either download a ready package:
dget <nowiki>http://http.debian.net/debian/pool/main/d/dh-virtualenv/dh-virtualenv_0.10-1.dsc</nowiki>
export DIST=stretch
sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc
or if you are developing a package and are in the package directory:
DIST=stretch pdebuild
Feel free to change stretch for the distribution of your choice from the ones above


apt-get install build-essential git-buildpackage cowbuilder
== Architectures ==
ARCH=amd64, or ARCH=i386 is supported if you feel like building for different architecture versions. There is no support for other architectures


Done.
== Debugging ==
By default, if the build fails, a hook is executed, providing the user with a shell allowing them to debug the build further. If that's not desired, there's a variable that can be defined to avoid that behavior. Example:
SHELL_ON_FAILURE=no pdebuild
If you reach the conclusion that your build fails because of some effort to write to HOME, and fixing the software to not do that is unfeasible, then you can set
BUILD\_HOME = /build
in .pbuilderrc or /etc/pbuilderrc


== Create images ==
== Additional Repos ==
=== Wikimedia repos ===
Aside from sid, the rest of the distributions allow for satisfying build time dependencies via the Wikimedia repos.


TODO Configure pbuilder:
To use packages from the Wikimedia repos to satisfy build dependencies during building you can use <code>WIKIMEDIA=yes</code>. There is also the approach of appending -wikimedia to the DIST variable and pbuilder will do what you want.


/root/.pbuilderrc
Examples:
Is APTCACHE really needed ?
<syntaxhighlight lang="shell">
$ DIST=stretch-wikimedia pdebuild
$ WIKIMEDIA=yes DIST=stretch pdebuild
</syntaxhighlight>
The commands above are equivalent and will both build a package for the stretch distribution using the Wikimedia apt repository.


== Stock Debian Jessie ==
=== Wikimedia Experimental repos ===


cowbuilder --create --distribution=jessie --debootstrapopts --variant=buildd
We can also add the wikimedia experimental repo using <code>WIKIMEDIA_EXPERIMENTAL=yes</code> in addtion to <code>WIKIMEDIA=yes</code>
<syntaxhighlight lang="shell">
$ WIKIMEDIA_EXPERIMENTAL=yes DIST=stretch-wikimedia pdebuild
$ WIKIMEDIA_EXPERIMENTAL=yes WIKIMEDIA=yes DIST=stretch pdebuild
</syntaxhighlight>


It takes the Debian mirror from your sources.list and install it in a chroot under <tt>/var/cache/pbuilder/base-jessie.cow</tt>.  That will be known as the distribution <tt>jessie</tt>More options are available via <tt>man cowbuilder</tt> and the underlying command being used <tt>man pbuilder</tt>.
=== Backports repos ===
Packages from the Debian backports repositories can be used to satisfy dependencies as well. To use the backports repository for the distribution selected (e.g. stretch-backports), use either of:
DIST=stretch BACKPORTS=yes pdebuild
  DIST=stretch-backports pdebuild


== Debian Jessie Wikimedia ==
=== Archived suites ===
Backports is only available for the regular lifetime of a Debian release, not during the LTS stage. After that packages should ideally be rebuilt using internally built components. If this isn't possible for some reason, the ARCHIVE\_BACKPORTS hook can be used to pull in the backports repository from archive.debian.org.
DIST=buster ARCHIVE\_BACKPORTS=yes pdebuild


Wikimedia uses its own apt repositories which comes with additional packages and backports. You will want to create another jessie flavor image. It would be similar but points to wikimedia and ensure the packages receive an higher priority. For a start, just copy:
=== Combining Wikimedia and Backports repos ===
Set both WIKIMEDIA and BACKPORTS:
DIST=stretch BACKPORTS=yes WIKIMEDIA=yes pdebuild
When using a distribution suffix, the other repo must be enabled via an environment variable. The following examples are equivalent:
DIST=stretch-backports WIKIMEDIA=yes pdebuild
DIST=stretch-wikimedia BACKPORTS=yes pdebuild


cp -a /var/cache/pbuilder/base-jessie.cow /var/cache/pbuilder/base-jessie-wikimedia.cow
== Additional Components ==


We will further tweak that image by login into it and having our modifications saved back to disk:
If you need to add a specific WMF component to the build environment then you can use the environment variable <code>COMPONENT</code>, you will also need to set WIKIMEDIA=yes e.g.
<syntaxhighlight lang=shell>
$ COMPONENT=component/puppet7 WIKIMEDIA=yes DIST=bullseye pdebuild
</syntaxhighlight>


cowbuilder --login --basepath /var/cache/pbuilder/base-jessie-wikimedia.cow --save-after-login
as well as this we also have a number of environment variables for specific combination of components to allow building common packages components


You now have a prompt in this image, add in the Wikimedia repository pinned to receive priority:
=== Spicerack ===
use the <code>SPICERACK</code> environment variable


echo deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty \
<syntaxhighlight lang=shell>
  > /etc/apt/sources.list.d/wikimedia.list
$ SPICERACK=yes pdebuild
echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" \
</syntaxhighlight>
  > /etc/apt/preferences.d/wikimedia


Since our packages are signed with GPG, you will want to grab the gpg keys we use and verify them. To do so:
=== Cergen ===
use the <code>CERGEN</code> environment variable, you also need to specify <code>DIST=buster</code> as this package only works on buster


apt-get install wget
<syntaxhighlight lang=shell>
wget http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg
$ CERGEN=yes DIST=buster pdebuild
apt-key add wikimedia-archive-keyring.gpg
</syntaxhighlight>
apt-get update


You will want to clear out some of the modifications you just made:


dpkg --purge wget
=== Java 8 ===
apt-get autoremove --purge
use the <code>JDK8</code> environment variable
rm wikimedia-archive-keyring.gpg


And upgrade the image:
<syntaxhighlight lang=shell>
$ JDK8=yes pdebuild
</syntaxhighlight>


apt-get upgrade
=== Opensearch 1.0 ===
use the <code>OPENSEARCH1</code> environment variable


exit / Ctrl+d and cowbuilder will save the result of your modification to the base-jessie-wikimedia.cow image.
<syntaxhighlight lang=shell>
$ OPENSEARCH1=yes pdebuild
</syntaxhighlight>


== Ubuntu images ==
=== PHP 7.2 ===
use the <code>PHP72</code> environment variable


TODO
<syntaxhighlight lang=shell>
$ PHP72=yes pdebuild
</syntaxhighlight>


    apt-get install ubuntu-archive-keyring
=== PHP 7.4 ===
    # We need universe!
use the <code>PHP74</code> environment variable
    cowbuilder --create \
        --basepath /packaging/base-trusty.cow \
        --mirror http://nova.clouds.archive.ubuntu.com/ubuntu/ \
        --distribution trusty \
        --components 'main universe' \
        --debootstrapopts --variant=buildd \
        --debootstrapopts --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg \


  echo 'deb http://apt.wikimedia.org/wikimedia precise-wikimedia main universe thirdparty' > /etc/apt/sources.list.d/wikimedia.list
<syntaxhighlight lang=shell>
  echo -e "Package: *\nPin: release o=Wikimedia\nPin-Priority: 1001" > /etc/apt/preferences.d/wikimedia
$ PHP74=yes pdebuild
</syntaxhighlight>


Probably need to get trusty updates as well.
=== VP 9 ===
use the <code>VP9</code> environment variable


=== Result ===
<syntaxhighlight lang=shell>
$ VP9=yes pdebuild
</syntaxhighlight>


An example result of the above steps:
== Using built packages as dependencies ==
By default, cowbuilder will always prefer upstream packages to packages you have built. If you have built a new version of a dependency, you typically want to use that version rather than the one provided by upstream. To force cowbuilder to use packages in the result directory, use:
APT_USE_BUILT=yes sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc


<pre>
=== Adding packages from a local directory ===
$ ls -1d /var/cache/pbuilder/*cow
The following is useful if you only need a specific package to satisfy the build dependencies.  you need to set <code>APT_USE_BUILT=yes</code> and <code>BUILDRESULT</code> pointing to the directory with the deb files e.g.
base-jessie.cow
base-jessie-wikimedia.cow
base-precise.cow
base-precise-wikimedia.cow
base-trusty.cow
base-trusty-wikimedia.cow
$
</pre>


This let you build a package against either the stock or the Wikimedia flavors for all distributions in use.
<syntaxhighlight lang="shell">
$ APT_USE_BUILT=yes BUILDRESULT=/home/jbond/debs/build DIST=bullseye pdebuild
</syntaxhighlight>


== git-buildpackage ==
git-pbuilder can be used by git-buildpackage to leverage all of the above but instead of DIST and ARCH you need to use --git-dist=$DIST and --git-arch=$ARCH.
WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=stretch
-sa is being used to enforce the original tarball to be included in the .changes file which is a requirement for Wikimedia reprepro.


= git-buildpackage =
The --git-no-pbuilder-autoconf/GIT\_PBUILDER\_AUTOCONF=no tells git-pbuilder to forego all attempts to discover the base path, tarball, or configuration file to set up the pbuilder options but rather instead rely on the settings in .pbuilderrc


= Maintaining patch series =
You can make it a default by editing your ~/.gbp.conf:
[buildpackage]
pbuilder = True
pbuilder-autoconf = False
dist = buster
Without "dist = buster" gbp will build for sid by default.


Placeholder for <tt>gbp pq</tt>.
== Results ==
The resulting deb files should be in /var/cache/pbuilder/result/${DIST}-${ARCH} like:
/var/cache/pbuilder/result/stretch-amd64/
 
== Notes ==
If you are getting confused over the naming of pbuilder/cowbuilder, here's some info to help you. pbuilder is the actual base software, cowbuilder is an extension to allow pbuilder to use COW (copy on write) instead of slow .tar.gz base files. For all intents and purposes this should be transparent to you as cowbuilder is the default pbuilder builder.
 
== Networking ==
cowbuilder/pbuilder block networking using Linux namespaces. Technically speaking an *unshare* is done in those environments, but the effect is that you can expect networking to not work.
 
If your package requires internet access to build successfully, it will not work.
 
First, try to fix the package. Packages in general should not require internet access to be built for a variety of reasons which will not be explained here.
 
If that is impossible/undesirable, then set
USENETWORK=yes
in /etc/pbuilderrc or ~/.pbuilderrc can be used to override that behaviour. Make sure that the building host has internet access though, or else your change will not be useful
 
== Cloud VPS ==
Apply <tt>role::labs::lvm::srv</tt> to get additional disk space at /srv.
Apply <tt>role::package::builder</tt>
 
[[Category:Package management]]
[[Category:SRE Infrastructure Foundations]]

Latest revision as of 12:37, 5 August 2022

Documentation available in: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/

Introduction

A module used to populate a Debian/Ubuntu package building environment. Meant to be used in the Wikimedia environment but could be adapted for other environments as well.

Setting it up

Include the package\_builder class in your machine. That can be done with whatever ENC you have puppet working with but in Wikimedia Labs you should create a puppet group, then add the class in the puppet group and just configure your VM with that class

After puppet is done you will have a number of pristine cowbuilder environments. Those would be:

  • stretch-amd64
  • buster-amd64
  • bullseye-amd64
  • sid-amd64

Building packages

You just either download a ready package:

dget http://http.debian.net/debian/pool/main/d/dh-virtualenv/dh-virtualenv_0.10-1.dsc
export DIST=stretch
sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc

or if you are developing a package and are in the package directory:

DIST=stretch pdebuild

Feel free to change stretch for the distribution of your choice from the ones above

Architectures

ARCH=amd64, or ARCH=i386 is supported if you feel like building for different architecture versions. There is no support for other architectures

Debugging

By default, if the build fails, a hook is executed, providing the user with a shell allowing them to debug the build further. If that's not desired, there's a variable that can be defined to avoid that behavior. Example:

SHELL_ON_FAILURE=no pdebuild

If you reach the conclusion that your build fails because of some effort to write to HOME, and fixing the software to not do that is unfeasible, then you can set

BUILD\_HOME = /build

in .pbuilderrc or /etc/pbuilderrc

Additional Repos

Wikimedia repos

Aside from sid, the rest of the distributions allow for satisfying build time dependencies via the Wikimedia repos.

To use packages from the Wikimedia repos to satisfy build dependencies during building you can use WIKIMEDIA=yes. There is also the approach of appending -wikimedia to the DIST variable and pbuilder will do what you want.

Examples:

$ DIST=stretch-wikimedia pdebuild
$ WIKIMEDIA=yes DIST=stretch pdebuild

The commands above are equivalent and will both build a package for the stretch distribution using the Wikimedia apt repository.

Wikimedia Experimental repos

We can also add the wikimedia experimental repo using WIKIMEDIA_EXPERIMENTAL=yes in addtion to WIKIMEDIA=yes

$ WIKIMEDIA_EXPERIMENTAL=yes DIST=stretch-wikimedia pdebuild
$ WIKIMEDIA_EXPERIMENTAL=yes WIKIMEDIA=yes DIST=stretch pdebuild

Backports repos

Packages from the Debian backports repositories can be used to satisfy dependencies as well. To use the backports repository for the distribution selected (e.g. stretch-backports), use either of:

DIST=stretch BACKPORTS=yes pdebuild
DIST=stretch-backports pdebuild

Archived suites

Backports is only available for the regular lifetime of a Debian release, not during the LTS stage. After that packages should ideally be rebuilt using internally built components. If this isn't possible for some reason, the ARCHIVE\_BACKPORTS hook can be used to pull in the backports repository from archive.debian.org.

DIST=buster ARCHIVE\_BACKPORTS=yes pdebuild

Combining Wikimedia and Backports repos

Set both WIKIMEDIA and BACKPORTS:

DIST=stretch BACKPORTS=yes WIKIMEDIA=yes pdebuild

When using a distribution suffix, the other repo must be enabled via an environment variable. The following examples are equivalent:

DIST=stretch-backports WIKIMEDIA=yes pdebuild
DIST=stretch-wikimedia BACKPORTS=yes pdebuild

Additional Components

If you need to add a specific WMF component to the build environment then you can use the environment variable COMPONENT, you will also need to set WIKIMEDIA=yes e.g.

$ COMPONENT=component/puppet7 WIKIMEDIA=yes DIST=bullseye pdebuild

as well as this we also have a number of environment variables for specific combination of components to allow building common packages components

Spicerack

use the SPICERACK environment variable

$ SPICERACK=yes pdebuild

Cergen

use the CERGEN environment variable, you also need to specify DIST=buster as this package only works on buster

$ CERGEN=yes DIST=buster pdebuild


Java 8

use the JDK8 environment variable

$ JDK8=yes pdebuild

Opensearch 1.0

use the OPENSEARCH1 environment variable

$ OPENSEARCH1=yes pdebuild

PHP 7.2

use the PHP72 environment variable

$ PHP72=yes pdebuild

PHP 7.4

use the PHP74 environment variable

$ PHP74=yes pdebuild

VP 9

use the VP9 environment variable

$ VP9=yes pdebuild

Using built packages as dependencies

By default, cowbuilder will always prefer upstream packages to packages you have built. If you have built a new version of a dependency, you typically want to use that version rather than the one provided by upstream. To force cowbuilder to use packages in the result directory, use:

APT_USE_BUILT=yes sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc

Adding packages from a local directory

The following is useful if you only need a specific package to satisfy the build dependencies. you need to set APT_USE_BUILT=yes and BUILDRESULT pointing to the directory with the deb files e.g.

$ APT_USE_BUILT=yes BUILDRESULT=/home/jbond/debs/build DIST=bullseye pdebuild

git-buildpackage

git-pbuilder can be used by git-buildpackage to leverage all of the above but instead of DIST and ARCH you need to use --git-dist=$DIST and --git-arch=$ARCH.

WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=stretch

-sa is being used to enforce the original tarball to be included in the .changes file which is a requirement for Wikimedia reprepro.

The --git-no-pbuilder-autoconf/GIT\_PBUILDER\_AUTOCONF=no tells git-pbuilder to forego all attempts to discover the base path, tarball, or configuration file to set up the pbuilder options but rather instead rely on the settings in .pbuilderrc

You can make it a default by editing your ~/.gbp.conf:

[buildpackage]
pbuilder = True
pbuilder-autoconf = False
dist = buster

Without "dist = buster" gbp will build for sid by default.

Results

The resulting deb files should be in /var/cache/pbuilder/result/${DIST}-${ARCH} like:

/var/cache/pbuilder/result/stretch-amd64/

Notes

If you are getting confused over the naming of pbuilder/cowbuilder, here's some info to help you. pbuilder is the actual base software, cowbuilder is an extension to allow pbuilder to use COW (copy on write) instead of slow .tar.gz base files. For all intents and purposes this should be transparent to you as cowbuilder is the default pbuilder builder.

Networking

cowbuilder/pbuilder block networking using Linux namespaces. Technically speaking an *unshare* is done in those environments, but the effect is that you can expect networking to not work.

If your package requires internet access to build successfully, it will not work.

First, try to fix the package. Packages in general should not require internet access to be built for a variety of reasons which will not be explained here.

If that is impossible/undesirable, then set

USENETWORK=yes

in /etc/pbuilderrc or ~/.pbuilderrc can be used to override that behaviour. Make sure that the building host has internet access though, or else your change will not be useful

Cloud VPS

Apply role::labs::lvm::srv to get additional disk space at /srv. Apply role::package::builder