You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Reprepro is a tool for managing APT repositories.
Reprepro is able to manage multiple repositories for multiple distribution versions and one package pool. It can process updates from an
incoming directory, copy package (references) between distribution versions, list all packages and/or package versions available in the repository, etc.
Reprepro maintains an internal database (a .DBM file) of the contents of the repository, which makes it quite fast and efficient.
It's installed from the Debian package
reprepro, and is configured using the files in
might be quite useful. One example is https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/admin/files/home/akosiaris/.variables;e86a9f707eadfef8bbfe23e314324f13750a1e36$3 and using
sudo -E reprepro <etc..etc>
to make life better.
This section explains the most commonly needed actions/tasks involving reprepro. reprepro is running on carbon.wikimedia.org
List all package versions in the repositories
For a given package name, use
reprepro ls packagename
# reprepro ls puppet puppet | 2.6.1-0ubuntu1~ppa1~hardy3 | hardy-wikimedia | amd64, source puppet | 2.6.1-0ubuntu1~ppa1~lucid1 | lucid-wikimedia | amd64, source
This shows that there are two different builds of the same package version in the repositories hardy-wikimedia and lucid-wikimedia. There is clearly no puppet package in the karmic-wikimedia repository.
To see all packages in a given distribution, use
reprepro list distribution-name
To find all packages in all repositories, use
or a variant thereof (see
reprepro help or
Building an unmodified third-party package for import
- set up a Labs instance for building and select the role::package::builder role. You could use the VMs in the packaging project.
- Alternatively if you are ops, just logging into copper.eqiad.wmnet. It is set up in exactly the same way but is faster and is already set up.
- fetch the package source to that instance and review it. dget, wget, curl are all valid options, dget is probably the faster.
- build the package for the target distribution using https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/README.md$30
- copy the build to the repo server (i.e. host with the install-server role, carbon on 20150414). On copper there is an rsync server that can be used directly from carbon. Example sudo rsync -va copper.eqiad.wmnet::pbuilder-result/trusty-amd64/apertium* .
- proceed with reprepro as documented below
Building an patched package for import
(e.g. when backporting a package from a more recent Debian release)
- import the package into git (usually operations/debs/SOURCEPACKAGENAME)
- apply your patch(es)
- set up a Labs instance for building and select the role::package::builder role.
- Alternatively if you are ops, just logging into copper.eqiad.wmnet. It is set up in exactly the same way but is faster and is already set up
- check out your package on the labs host
- copy the build to the repo server (i.e. host with the install-server role, carbon on 20150514).On copper there is an rsync server that can be used directly from carbon. Example sudo rsync -va copper.eqiad.wmnet::pbuilder-result/trusty-amd64/apertium* .
- proceed with reprepro as documented below
Automatically import files from an
Reprepro can automatically import packages from an upload directory, as long as all the package fields are setup correctly with the right distribution and component names. It's also vital that the
.changes files are present. When all these conditions are met, and all these files have been uploaded to
/srv/wikimedia/incoming (e.g. using
dupload), you can use:
# reprepro processincoming default
It uses the rules defined in the file
If the package is rejected by reprepro because one of the package control fields are wrong, or you want to override them for some other reason, use an override file (see below).
It's best to check whether the
/srv/wikimedia/incoming/ directory is empty after using
procesincoming, because reprepro should have moved/deleted all imported files. Any remaining files have not been processed.
It's always best to have reprepro fully manage all package aspects using the
.changes that was created during the build of the package (e.g. using Pbuilder). When the
.changes file is present along with all files list therein, reprepro can handle it all with the
reprepro include command:
# reprepro -C component-name include wikimedia-distribution-name path-to-.changes-file
# reprepro -C main include hardy-wikimedia php5-apc_3.1.3p1-1wm1_amd64.changes
For component, use the following as a guide:
- for Wikimedia native packages, as well as Debian/Ubuntu packages that have had source-modifications
- for existing Debian/Ubuntu packages that just have been recompiled/backported for the given distribution.
For distribution, use the distribution that the package has been compiled for, and under. Usually, any given compiled package should be for one distribution only, e.g. hardy-wikimedia OR lucid-wikimedia. This should match the field in the package's Changelog. Only in special circumstances a given package can be used under multiple distribution versions, .e.g. if the package contains only scripts (no binaries). Only do this if you've tested it and are sure this is working. In this case, include the package in the oldest distribution version, and then copy the package references using
reprepro copy (see below). If your package was
specifically built for wikimedia and does not have a distribution of
*-wikimedia listed in the
.changes file, then you should force reprepro to accept a
*-wikimedia distribution. (You'll probably want
precise-wikimedia). Add the
--ignore=wrongdistribution flag to the reprepro command to do so.
.changes file is available, for example because you didn't build the package yourself, you can use
reprepro includedsc and
# reprepro -C universe includedsc lucid-wikimedia varnish_2.1.2-1.dsc # reprepro -C universe includedeb lucid-wikimedia varnish_2.1.2-1_amd64.deb # reprepro -C universe includedeb lucid-wikimedia libvarnish1_2.1.2-1_amd64.deb # reprepro -C universe includedeb lucid-wikimedia libvarnish-dev_2.1.2-1_amd64.deb
Be aware that reprepro will remove older versions of packages without asking. They are no longer available in the pool (
/srv/wikimedia/pool) either. However,
/srv/wikimedia/ is backed up using Amanda on tridge, and many packages should be available in subversion as well.
A given package can be removed from a distribution using
# reprepro remove distribution-name package-name
# reprepro remove hardy-wikimedia facter
Using the override file
When we are building our own packages, we should make sure that all control fields (such as the distribution name, component, priority etc.) are set correctly. Please rebuild your package if not.
However, occasionally it is necessary to override fields on a previously built package, which we don't want to modify the source of and/or rebuild. Ubuntu often does the same, and just retrieves packages from Debian Unstable and overrides a few fields using an override file.
We have an override file as well, in
/srv/wikimedia/conf/deb-override. It's format is:
# packagename fieldname newvalue
As an example, our Varnish packages are coming straight from Debian Unstable (like in Ubuntu), and can be imported fine into Lucid as long as we override some package fields:
varnish Distribution lucid libvarnish1 Distribution lucid libvarnish-dev Distribution lucid
Copying between distributions
In some cases, notably when no compiled binaries are involved, a given package build can be used for multiple distribution versions. This is for example the case with most native Wikimedia packages, which just contain (shell) scripts and can easily support multiple distribution versions (e.g. Hardy to Lucid). It would be a waste of time and work to make separate builds for these packages, so we include only one copy of these packages into multiple distributions.
To do this, build the package for the oldest distribution that is supported, e.g.
hardy-wikimedia. Then, import it into this distribution:
# reprepro -C main include hardy-wikimedia wikimedia-base wikimedia-base_0.26_amd64.changes # reprepro ls wikimedia-base wikimedia-base | 0.26 | hardy-wikimedia | amd64, source
Then, copy it to the other supported distributions, e.g. karmic-wikimedia and lucid-wikimedia:
# reprepro copy karmic-wikimedia hardy-wikimedia wikimedia-base # reprepro copy lucid-wikimedia hardy-wikimedia wikimedia-base # reprepro ls wikimedia-base wikimedia-base | 0.26 | hardy-wikimedia | amd64, source wikimedia-base | 0.26 | karmic-wikimedia | amd64, source wikimedia-base | 0.26 | lucid-wikimedia | amd64, source
Updating external repositories
Reprepro has the ability to pull packages from other APT repositories automatically, this has added benefits like verifying signatures, easy management and so on. It is configured via conf/updates configuration file.
To check which updates are available:
# reprepro --restrict cassandra checkupdate Calculating packages to get... Updates needed for 'jessie-wikimedia|thirdparty|source': 'cassandra': '2.1.5' will be upgraded to '2.1.6' (from 'cassandra'): files needed: pool/thirdparty/c/cassandra/cassandra_2.1.6.dsc pool/thirdparty/c/cassandra/cassandra_2.1.6.orig.tar.gz pool/thirdparty/c/cassandra/cassandra_2.1.6.diff.gz ...
Remember to use --restrict to limit your update only the packages you are interested in, otherwise all available updates will be brought in!
To pull in the updates for real:
# reprepro --restrict cassandra update Calculating packages to get... Getting packages... Installing (and possibly deleting) packages... Exporting indices... Deleting files no longer referenced...
If signing fails
If you get errors such as:
Error: gpgme created no signature! This most likely means gpg is confused or produces some error libgpgme is not able to understand.
Make sure you are running as root AND you have exported its environment variables to point to the right gpg keyring (at least until other method of signing is setup). This usually means:
export REPREPRO_BASE_DIR=/srv/wikimedia export GNUPGHOME=/root/.gnupg sudo -E ...
- Debian Reprepro (Upstream documentation and source code)