Reprepro

Reprepro is a tool for managing APT repositories.

Reprepro is able to manage multiple repositories for multiple distribution versions and one package pool. It can process updates from an incoming directory, copy package (references) between distribution versions, list all packages and/or package versions available in the repository, etc.

Reprepro maintains an internal database (a .DBM file) of the contents of the repository, which makes it quite fast and efficient.

It's installed from the Debian package reprepro, and is configured using the files in /srv/wikimedia/conf/. Running

# export REPREPRO_BASE_DIR=/srv/wikimedia
# export GNUPGHOME=/root/.gnupg

might be quite useful. One example is https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/admin/files/home/akosiaris/.variables;e86a9f707eadfef8bbfe23e314324f13750a1e36$3 and using

  sudo -E reprepro <etc..etc>

to make life better.

HOWTO

This section explains the most commonly needed actions/tasks involving reprepro. reprepro is running on install1002.wikimedia.org.

List all package versions in the repositories

For a given package name, use

reprepro ls PACKAGE_NAME

For example:

# reprepro ls puppet
puppet | 4.8.2-5~trusty1 |  trusty-wikimedia | amd64, i386, source
puppet |  4.8.2-5~bpo8+1 |  jessie-wikimedia | amd64, i386, source
puppet |  3.8.5-2~bpo8+2 |  jessie-wikimedia | amd64, i386, source
puppet |         4.8.2-5 | stretch-wikimedia | amd64, i386, source

This shows that there are two different builds of the same package version in the repositories hardy-wikimedia and lucid-wikimedia. There is clearly no puppet package in the karmic-wikimedia repository.

To see all packages in a given distribution, use

reprepro list DISTRIBUTION_NAME

To find all packages in all repositories, use

reprepro dumpreferences

or a variant thereof (see reprepro --help or man reprepro).

Building an unmodified third-party package for import

1. set up a Labs instance for building and select the role::package::builder role. You could use the VMs in the packaging project.
2. Alternatively if you are ops, just logging into boron.eqiad.wmnet. It is set up in exactly the same way but is faster and is already set up.
3. fetch the package source to that instance and review it. dget, wget, curl are all valid options, dget is probably the faster.
4. build the package for the target distribution using https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/README.md$30
5. copy the build to the repo server (i.e. host with the install-server roles, install1002/2002 on 20170214). On boron there is an rsync server that can be used directly from install1002. Example sudo rsync -va boron.eqiad.wmnet::pbuilder-result/trusty-amd64/apertium* .
6. proceed with reprepro as documented below

Building a patched package for import

(e.g. when backporting a package from a more recent Debian release)

1. import the package into git (usually operations/debs/SOURCEPACKAGENAME)
2. apply your patch(es)
3. set up a Labs instance for building and select the role::package::builder role.
4. Alternatively if you are ops, just logging into boron.eqiad.wmnet. It is set up in exactly the same way but is faster and is already set up
5. check out your package on the labs host
6. build the package for the target distribution using https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/README.md$30
7. copy the build to the repo server (i.e. host with the install-server roles, install1002/2002 on 20170214).On boron there is an rsync server that can be used directly from install1002. Example sudo rsync -va boron.eqiad.wmnet::pbuilder-result/trusty-amd64/apertium* .
8. proceed with reprepro as documented below

Automatically import files from an incoming/ directory

Reprepro can automatically import packages from an upload directory, as long as all the package fields are setup correctly with the right distribution and component names. It's also vital that the .changes files are present. When all these conditions are met, and all these files have been uploaded to /srv/wikimedia/incoming (e.g. using dupload), you can use:

# reprepro processincoming default

It uses the rules defined in the file /srv/wikimedia/conf/incoming

If the package is rejected by reprepro because one of the package control fields are wrong, or you want to override them for some other reason, use an override file (see below).

It's best to check whether the /srv/wikimedia/incoming/ directory is empty after using procesincoming, because reprepro should have moved/deleted all imported files. Any remaining files have not been processed.

Importing packages

You will need three things:

1. Component
   1. main: for Wikimedia native packages, as well as Debian/Ubuntu packages that have had source-modifications
   2. universe: for existing Ubuntu packages that just have been recompiled or backported for the given distribution.
2. Distribution
   1. Usually: stretch-wikimedia, jessie-wikimedia, or trusty-wikimedia. This is the distribution that the package has been compiled for, and under. This should match the field in the package's Changelog.
   2. If your package was specifically built for Wikimedia and does not have a distribution of CODENAME-wikimedia listed in the changes file, then you should force reprepro to accept a CODENAME-wikimedia distribution. Add --ignore=wrongdistribution flag to the reprepro command to do so.
3. Changes file
   1. If building on the packaging server, the changes file is most likely in /var/cache/pbuilder/result/CODENAME-amd64/ along with the dsc, buildinfo, tar.xz, deb, and orig.tar.gz.
   2. You will want to bring all of these files over to the reprepro server.

It's best to have reprepro fully manage all package aspects using the changes file that was created during the build of the package (e.g. using Pbuilder). When the changes file is present along with all files list therein, reprepro can handle it all with:

# reprepro -C COMPONENT include DISTRIBUTION CHANGES_FILE

For example:

# reprepro -C main include stretch-wikimedia nagios-nrpe-server_3.0.1-3+deb9u1+wmf1_amd64.changes

When no changes file is available, for example because you didn't build the package yourself, you can use includedsc and includedeb:

# reprepro -C universe includedsc lucid-wikimedia varnish_2.1.2-1.dsc
# reprepro -C universe includedeb lucid-wikimedia varnish_2.1.2-1_amd64.deb
# reprepro -C universe includedeb lucid-wikimedia libvarnish1_2.1.2-1_amd64.deb
# reprepro -C universe includedeb lucid-wikimedia libvarnish-dev_2.1.2-1_amd64.deb

Be aware that reprepro will remove older versions of packages without asking. They are no longer available in the pool (/srv/wikimedia/pool) either. However, /srv/wikimedia/ is backed up using Amanda on tridge, and many packages should be available in subversion as well.

Missing orig tarball

The original tarball should be listed in the changes file when package is built. Often times, this is accomplished with a build flag: dpkg-buildpackage -sa, debuild -sa, or pdebuild -- --debbuildopts -sa

If it is missing though, you can tell reprepro to ignore it and find it in the current working directory with the --ignore=missingfile flag.

Removing packages

A given binary package can be removed from a distribution using

# reprepro remove distribution-name package-name

For example:

# reprepro remove jessie-wikimedia facter

However, usually you want to remove all binary packages from a source package, that can be done with:

# reprepro removesrc distribution-name package-name

For example:

# reprepro removesrc trusty-wikimedia openjdk-8

Using the override file

When we are building our own packages, we should make sure that all control fields (such as the distribution name, component, priority etc.) are set correctly. Please rebuild your package if not.

However, occasionally it is necessary to override fields on a previously built package, which we don't want to modify the source of and/or rebuild. Ubuntu often does the same, and just retrieves packages from Debian Unstable and overrides a few fields using an override file.

We have an override file as well, in /srv/wikimedia/conf/deb-override. It's format is:

# packagename	fieldname	newvalue

As an example, our Varnish packages are coming straight from Debian Unstable (like in Ubuntu), and can be imported fine into Lucid as long as we override some package fields:

varnish		Distribution	lucid
libvarnish1	Distribution	lucid
libvarnish-dev	Distribution	lucid

Copying between distributions

In some cases it's necessary to copy a binary which gets reused in a different distribution, e.g. for Go packages (which are statically linked and usually hard/impossible to rebuild in older distro versions). Note that in contrast to typical conventions you first need to specify the destination and then the source!. In the following example we copy from stretch to buster:

# reprepro copy buster-wikimedia stretch-wikimedia prometheus-rsyslog-exporter
Exporting indices...
# reprepro ls prometheus-rsyslog-exporter
prometheus-rsyslog-exporter | 0.0.0+git20180118-1 |  jessie-wikimedia | amd64, source
prometheus-rsyslog-exporter | 0.0.0+git20180118-1 | stretch-wikimedia | amd64, source
prometheus-rsyslog-exporter | 0.0.0+git20180118-1 |  buster-wikimedia | amd64, source

Updating external repositories

Reprepro has the ability to pull packages from other APT repositories automatically, this has added benefits like verifying signatures, easy management and so on. It is configured via conf/updates configuration file managed via Puppet.

To check which updates are available:

  # reprepro --component thirdparty/tor checkupdate stretch-wikimedia
Calculating packages to get...
Updates needed for 'stretch-wikimedia|thirdparty/tor|source':
'tor': '0.3.3.9-1~d90.stretch+1' will be upgraded to '0.3.5.7-1~d90.stretch+1' (from 'tor-stretch'):
 files needed: pool/thirdparty/tor/t/tor/tor_0.3.5.7-1~d90.stretch+1.dsc pool/thirdparty/tor/t/tor/tor_0.3.5.7.orig.tar.gz pool/thirdparty/tor/t/tor/tor_0.3.5.7-1~d90.stretch+1.diff.gz
 Updates needed for 'stretch-wikimedia|thirdparty/tor|amd64':
'tor': '0.3.3.9-1~d90.stretch+1' will be upgraded to '0.3.5.7-1~d90.stretch+1' (from 'tor-stretch'):
 files needed: pool/thirdparty/tor/t/tor/tor_0.3.5.7-1~d90.stretch+1_amd64.deb
'tor-dbgsym': '0.3.3.9-1~d90.stretch+1' will be upgraded to '0.3.5.7-1~d90.stretch+1' (from 'tor-stretch'):
 files needed: pool/thirdparty/tor/t/tor/tor-dbgsym_0.3.5.7-1~d90.stretch+1_amd64.deb
'tor-geoipdb': '0.3.3.9-1~d90.stretch+1' will be upgraded to '0.3.5.7-1~d90.stretch+1' (from 'tor-stretch'):
 files needed: pool/thirdparty/tor/t/tor/tor-geoipdb_0.3.5.7-1~d90.stretch+1_all.deb

You can also get an overview of all packages currently pending for a distro using 'checkupdate':

reprepro checkupdate stretch-wikimedia

To pull in the updates for real:

 # reprepro --noskipold  --component thirdparty/php72 update stretch-wikimedia
Calculating packages to get...
Getting packages...
Installing (and possibly deleting) packages...
Exporting indices...
Deleting files no longer referenced...

Adding a new external repository

When adding a new external repository, you need to first add a new definition to modules/aptrepo/files/updates in puppet.git.

When puppet has been run on install1002.wikimedia.org, you need to import the GPG key used to sign the new repository to the root user's home:

sudo -H bash
gpg --keyserver hkp://pool.sks-keyservers.net --recv-key $GPGKEYFINGERPRINT

Updating where Pull rules are used

If a Pull rule is configured between internal distributions, the checkpull and pull commands needs to be used instead to copy packages. For example, thirdparty/cloudera is pulled from an external repository to jessie-wikimedia and then a Pull rule is present for stretch-wikimedia, so an upgrade should follow this procedure: Analytics/Systems/Cluster/Hadoop/Administration#Updating Cloudera Packages.

If signing fails

If you get errors such as:

Error: gpgme created no signature!
This most likely means gpg is confused or produces some error libgpgme is
not able to understand.

Make sure you are running as root AND you have exported its environment variables to point to the right gpg keyring (at least until other method of signing is setup). This usually means:

sudo -i

or, better,

export REPREPRO_BASE_DIR=/srv/wikimedia
export GNUPGHOME=/root/.gnupg
sudo -E ...

Additional info

Some additional bits you may find useful.

dupload

To upload .changes files with dupload you will likely need a configuration like this in /etc/dupload.conf:

[...]
$cfg{'wmf'} = {
    fqdn => 'install1002.wikimedia.org',
    method => 'scp',
    incoming => '/srv/wikimedia/incoming',
    login => 'yourusername',
};
[...]

And then, you will be able to upload to incoming using a command like:

$ dupload --to wmf ../pkg_version_amd64.changes

External links

• Debian Reprepro (Upstream documentation and source code)