You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Debian Glue

From Wikitech-static
Revision as of 12:06, 9 September 2016 by imported>Hashar (bunch of basic documentation)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

We have Jenkins jobs able to build a Debian package in a cowbuilder image. The job uses a serie of shell wrapper which streamline the environment setup and debian toolchain: http://jenkins-debian-glue.org/.

As of September 2016, the jobs are running on Jessie jenkins agents. cobuilder images are provisioned by our Puppet modules package_builder which:

  • create images for the distribution we care about (ex: trusty, jessie, unstable)
  • auto update the images on a daily basis
  • provides hook to support injecting apt.wikimedia.org components and the jessie-backports

For Gerrit repositories having Debian packages, the Jenkins job is registered in Zuul workflow to trigger:

integration/config.git /zuul/layout.yaml
  - name: operations/debs/contenttranslation/apertium
      test:
         - 'debian-glue'

Will cause any patchset proposed to that Gerrit repository to trigger the Jenkins job debian-glue. The job:

  • clone the repository
  • checkout the patch merged against tip of the branch
  • set distribution to the distribution mentioned in debian/changelog
  • invoke Jenkins Debian Glue.

Jenkins Debian Glue will set DIST based on distribution. Then trigger the build process in the matching cowbuilder image. When pbuilder is run, the Wikimedia hooks are invoked and whenever the debian/changelog distribution is suffixed with -wikimedia, Wikimedia more specific components (eg: thirdparty) will be added.

As of September 2016, setting BACKPORTS to inject the Ubuntu updates component or Debian backports components is NOT SUPPORTED.

Specific env variables can be injected by Zuul. This is done by altering a python script executed by Zuul whenever it triggers a job. Example as of September 2016:

integration/config.git /zuul/parameter_functions.py
if 'debian-glue' in job.name:
    # Always set the value to be safe (T144094)
    params['BUILD_TIMEOUT'] = 30  # minutes
    # Finely tweak jenkins-debian-glue parameters
    if params['ZUUL_PROJECT'] == 'integration/zuul':
        # Uses dh_virtualenv which needs access to pypy.python.org
        params['PBUILDER_USENETWORK'] = 'yes'
        params['DEB_BUILD_OPTIONS'] = 'nocheck'
    elif (params['ZUUL_PROJECT'] ==
            'operations/debs/contenttranslation/giella-sme'):
        # Heavy build T143546
        params['BUILD_TIMEOUT'] = 180  # minutes

Since the build is done with a sudo cowbuilder, each new environment variable has to be whitelisted in the sudo policy of the integration labs project.

Non exhaustive list as of September 2016:


Env Description
DEB_* ???
DIST Distribution used by cowbuilder, pbuilder and Wikimedia pbuilder hook
ARCH Architecture (i386, amd64)
BUILDRESULT Debian glue setting
distribution Debian glue setting. Set by the job to the distribution in debian/changelog
WORKSPACE Base directory of the Jenkins job. Set by Jenkins.
DEB_BUILD_OPTIONS  Defined by Debian Policy, let you change behavior of the build process if proper support is added in debian/rules. A typical example is to bypass tests: DEB_BUILD_OPTIONS=nocheck. If needed, must be injected by Zuul.

References