Debian Glue

We have Jenkins jobs able to build a Debian package in a cowbuilder image. The jobs use a serie of shell wrapper which streamline the environment setup and debian toolchain: http://jenkins-debian-glue.org/.

The jobs are running on dedicated Jenkins agents hosted on WMCS. They come with cowbuilder images provisioned by our Puppet modules package_builder which:

• create images for the distribution we care about (ex: trusty, jessie, unstable)
• auto update the images on a daily basis
• provides hook to support injecting apt.wikimedia.org components and the <release>-backports distribution (example: buster-backports).

For Gerrit repositories having Debian packages, the Jenkins job is registered in Zuul workflow to trigger:

integration/config.git /zuul/layout.yaml

  - name: operations/debs/contenttranslation/apertium
      test:
         - 'debian-glue'

It instructs CI to run the debian-glue job for any patchset proposed to that Gerrit repository. But it only triggers if the change touches a file under the debian directory. The job then:

• clones the repository
• checkouts the patch that has been merged by CI against the tip of the targeted branch
• sets distribution to the distribution mentioned in debian/changelog (see [#distributions] below).
• invokes Jenkins Debian Glue.

Jenkins Debian Glue will set DIST based on distribution. Then it triggers the build process in the matching cowbuilder image. When pbuilder is run, the Wikimedia hooks are invoked and whenever the debian/changelog distribution is suffixed with -wikimedia, Wikimedia more specific components (eg: thirdparty) will be added.

Specific env variables can be injected by Zuul. This is done by altering a python script executed by Zuul whenever it triggers a job. That can be used to set BACKPORTS which instruct our hooks to inject the release-backports components, tweak the build timeout with BUILD_TIMEOUT or pass DEB_BUILD_OPTIONS. Example as of May 2020:

integration/config.git /zuul/parameter_functions.py

    if 'debian-glue' in job.name:

        # XXX
        # When adding new paramters, make sure the env variable is added as an
        # env_keep in the sudo policy:
        # https://horizon.wikimedia.org/project/sudo/
        #

        if 'nocheck' in job.name:
            params['DEB_BUILD_OPTIONS'] = 'nocheck'
        if 'backports' in job.name:  # T173999
            params['BACKPORTS'] = 'yes'
        # Always set the value to be safe (T144094)
        params['BUILD_TIMEOUT'] = 30  # minutes
        # Finely tweak jenkins-debian-glue parameters
        if params['ZUUL_PROJECT'] == 'integration/zuul':
            # Uses dh_virtualenv which needs access to pypy.python.org
            params['PBUILDER_USENETWORK'] = 'yes'
        elif (params['ZUUL_PROJECT'] == 'operations/debs/varnish4'):
            # VTC tests take forever 
            params['BUILD_TIMEOUT'] = 60  # minutes
            params['DEB_BUILD_OPTIONS'] = 'parallel=12'
        elif (params['ZUUL_PROJECT']
              == 'operations/software/varnish/varnishkafka'):
            # needed for librdkafka1 >= 0.11.5
            params['BACKPORTS'] = 'yes'
        elif (params['ZUUL_PROJECT'] == 'operations/software/atskafka'):
            # needed by go build to access gopkg.in
            params['PBUILDER_USENETWORK'] = 'yes'
        elif (params['ZUUL_PROJECT'] == 'operations/debs/trafficserver'):
            # Building ATS takes a while
            params['BUILD_TIMEOUT'] = 60  # minutes
            # Backports needed on stretch for libbrotli-dev and a recent
            # debhelper version (>= 11)
            params['BACKPORTS'] = 'yes'
        elif (params['ZUUL_PROJECT']
              == 'operations/debs/contenttranslation/giella-sme'):
            # Heavy build T143546
            params['BUILD_TIMEOUT'] = 180  # minutes

Since the build is done with a sudo cowbuilder, each new environment variable has to be whitelisted in the sudo policy of the integration labs project.

Non exhaustive list as of September 2016:

Env	Description
DEB_*	???
DIST	Distribution used by cowbuilder, pbuilder and Wikimedia pbuilder hook
ARCH	Architecture (i386, amd64)
BUILDRESULT	Debian glue setting
distribution	Debian glue setting. Set by the job to the distribution in debian/changelog
WORKSPACE	Base directory of the Jenkins job. Set by Jenkins.
DEB_BUILD_OPTIONS	Defined by Debian Policy, let you change behavior of the build process if proper support is added in debian/rules. A typical example is to bypass tests: DEB_BUILD_OPTIONS=nocheck. If needed, must be injected by Zuul.

distributions

Jenkins debian glue parses the debian/changelog to find the target distribution. When the changelog entry targets UNRELEASED, CI picks the distribution from the previous changelog entry.

In some case ones want to set both WIKIMEDIA and BACKPORTS, but both can not be set in the changelog entry. Our convention is to suffix the distribution with -wikimedia in the changelog and then trigger the more specific job debian-glue-backports which cause CI to set BACKPORT=yes.

Some repository want a single branch to support multiple distributions. We thus have a set of jobs that hardcode the distribution and do not extract it from debian/changelog. Examples:

• debian-glue-stretch
• debian-glue-buster
• debian-glue-unstable

This is example for labs/toollabs which in zuul/layout.yaml has:

  - name: labs/toollabs
    test:
     # Single branch supporting multiple distributions T210780
      - debian-glue-unstable
      - debian-glue-stretch
      - debian-glue-buster
    gate-and-submit:
      - debian-glue-unstable
      - debian-glue-stretch
      - debian-glue-buster

References

• http://jenkins-debian-glue.org/
• Jenkins Debian Glue settings with documentation: http://jenkins-debian-glue.org/docs/
• Zuul workflow https://phabricator.wikimedia.org/diffusion/CICF/browse/master/zuul/layout.yaml
• Zuul python hook to inject env variables https://phabricator.wikimedia.org/diffusion/CICF/browse/master/zuul/parameter_functions.py
• Wikitech sudo policy: Special:NovaSudoer
• Definition of our Jenkins jobs using the Jenkins Job Builder DSL: https://phabricator.wikimedia.org/diffusion/CICF/browse/master/jjb/operations-debs.yaml