You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Debian Packaging: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Ayounsi
(Adding instruction to build a package for a new Debian version)
imported>Mainframe98
(7 intermediate revisions by 7 users not shown)
Line 1: Line 1:
{{TOC limit|4}}
These are runbooks related to Debian packaging for Wikimedia Foundation production.{{TOC limit|4}}


== Rebuilding a package ==
== Rebuilding a package ==
Line 7: Line 7:
Example: repackage <code>python-libthumbor</code> for debian stretch
Example: repackage <code>python-libthumbor</code> for debian stretch


1. Login to <code>boron.eqiad.wmnet</code>:
1. Login to <code>build2001.codfw.wmnet</code>:


2. As a user run the following to download <code>python-libthumbor</code>'s source package:
2. As a user run the following to download <code>python-libthumbor</code>'s source package:
  '''$ apt-get source libthumbor=1.3.2-0+wmf1'''
  $ apt-get source libthumbor=1.3.2-0+wmf1
3. We should now have 3 files and 1 directory in our working directory:
3. We should now have 3 files and 1 directory in our working directory:
   -rw-r--r--  1 user wikidev  15K Jul  6  2016 libthumbor_1.3.2.orig.tar.gz          -> source code of the upstream authors
   -rw-r--r--  1 user wikidev  15K Jul  6  2016 libthumbor_1.3.2.orig.tar.gz          -> source code of the upstream authors
Line 17: Line 17:
   drwxr-xr-x  6 user wikidev 4.0K Nov 20 14:11 libthumbor-1.3.2                      -> extracted source package
   drwxr-xr-x  6 user wikidev 4.0K Nov 20 14:11 libthumbor-1.3.2                      -> extracted source package
4. In the <code>libthumbor-1.3.2/debian</code> directory, we may find the following files:
4. In the <code>libthumbor-1.3.2/debian</code> directory, we may find the following files:
   '''changelog''' -> records all changes and defines the version of the package
   changelog -> records all changes and defines the version of the package
   '''compat''' -> a meta file used for automation of the package build
   compat -> a meta file used for automation of the package build
   '''control''' -> defines the binary package names and their relation to other packages (e.g. files needed to build and run the package)
   control -> defines the binary package names and their relation to other packages (e.g. files needed to build and run the package)
   '''copyright''' -> decribes the license and the copyright holders
   copyright -> decribes the license and the copyright holders
   '''docs''' -> this can be used to declare some files as documentation
   docs -> this can be used to declare some files as documentation
   '''patches''' -> a directory of code changes done to the original tarball
   patches -> a directory of code changes done to the original tarball
   '''rules''' -> defines the package build process
   rules -> defines the package build process
   '''source''' -> a directory defining some details of how the source package is defined
   source -> a directory defining some details of how the source package is defined
   '''watch''' -> a meta file which contains information where to look for new releases (i.e. where to download a newer source)
   watch -> a meta file which contains information where to look for new releases (i.e. where to download a newer source)
5. Go to <code>libthumbor-1.3.2</code>:
5. Go to <code>libthumbor-1.3.2</code>:
  '''$ libthumbor-1.3.2'''
  $ libthumbor-1.3.2
  '''$ dch -i'''
  $ dch -i


6. We should see something like the following:
6. We should see something like the following:
Line 55: Line 55:


8. Build!
8. Build!
  '''$ DIST=stretch pdebuild'''   -> (use packages from stretch repo)
  $ DIST=stretch pdebuild  -> (use packages from stretch repo)
  or  
  or  
  '''$ DIST=stretch-wikimedia pdebuild''' -> (include packages from stretch-wikimedia repo)
  $ DIST=stretch-wikimedia pdebuild  -> (include packages from stretch-wikimedia repo)
If building for a new Debian version do something like: <code>DIST=buster WIKIMEDIA=yes pdebuild --debbuildopts -sa</code>  
If building for a new Debian version do something like: <code>DIST=buster WIKIMEDIA=yes pdebuild --debbuildopts -sa</code>  


9. Find the build results:
9. Find the build results:
  user@boron:~'''$ ls  -la /var/cache/pbuilder/result/stretch-amd64/*thum*'''
  $ ls  -la /var/cache/pbuilder/result/stretch-amd64/*thum*
  -rw-r--r-- 1 user wikidev 5873 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/libthumbor_1.3.2-0+wmf1+stretch1_amd64.buildinfo
  -rw-r--r-- 1 user wikidev 5873 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/libthumbor_1.3.2-0+wmf1+stretch1_amd64.buildinfo
  -rw-r--r-- 1 user wikidev 1797 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/libthumbor_1.3.2-0+wmf1+stretch1_amd64.changes
  -rw-r--r-- 1 user wikidev 1797 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/libthumbor_1.3.2-0+wmf1+stretch1_amd64.changes
Line 70: Line 70:
=== Download source packages from snapshot.debian.org ===
=== Download source packages from snapshot.debian.org ===
If you are looking  to build a different version than the one provided by a source package, you can have a look at [https://snapshot.debian.org Debian snapshots]. Everything you need will be downloaded for you by pointing <code>dget</code> to the relevant <code>.dsc</code> file:
If you are looking  to build a different version than the one provided by a source package, you can have a look at [https://snapshot.debian.org Debian snapshots]. Everything you need will be downloaded for you by pointing <code>dget</code> to the relevant <code>.dsc</code> file:
  '''$ dget https://snapshot.debian.org/archive/debian-debug/20180826T210249Z/pool/main/libr/librsvg/librsvg_2.40.20-3.dsc'''
  $ dget https://snapshot.debian.org/archive/debian-debug/20180826T210249Z/pool/main/libr/librsvg/librsvg_2.40.20-3.dsc


=== Common problems ===
=== Common problems ===
Line 79: Line 79:
Adding <code>BACKPORTS=yes</code> to the beginning of your <code>pdebuild</code> invocation will often fix this.
Adding <code>BACKPORTS=yes</code> to the beginning of your <code>pdebuild</code> invocation will often fix this.


=== Misc notes ===
* If we are building a new package where we are the owners, like '''python-thumbor-wikimedia''', we should use <code>native</code> instead of <code>quilt</code> in <code>debian/source/format</code>
* If we are to use packages from eg '''php72''' component, this can be done by using <code>PHP72=yes</code> env var before building.
== Upload to Wikimedia Repo ==
== Upload to Wikimedia Repo ==
1. Login to <code> install1002.wikimedia.org</code>, create a directory and copy all files from boron
1. Login to <code>apt1001.wikimedia.org</code>, create a directory and copy all files from build2001
  '''$ ssh install1002.wikimedia.org'''
  $ ssh apt1001.wikimedia.org
  '''$ mkdir libthumbor; cd libthumbor'''
  $ mkdir libthumbor; cd libthumbor
  '''$ rsync rsync://boron.eqiad.wmnet/pbuilder-result/stretch-amd64/*thum* .'''
  $ rsync rsync://build2001.codfw.wmnet/pbuilder-result/stretch-amd64/*thum* .
2. Import!
2. Import!
  '''$ sudo -H bash'''
  $ sudo -H bash
  '''$ reprepro -C main include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes'''
  $ reprepro -C main include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes
Note: We split the archives sometimes, i.e. we add packages to components instead of <code>main</code>
Note: We split the archives sometimes, i.e. we add packages to components instead of <code>main</code>


3. Do not forget to '''!log uploads in -operations IRC channel''' !
3. Remember to log your uploads in #wikimedia-operations on IRC using  <code>!log</code>


4. ''Achievement Unlocked!''
4. ''Achievement Unlocked!''
Line 95: Line 98:
Sometimes we build packages for specific purposes thus we don't want them to be available under the <code>main</code> component.  
Sometimes we build packages for specific purposes thus we don't want them to be available under the <code>main</code> component.  
To create a new component, you simply need to prepare a patch for <code>puppet/modules/aptrepo/files/distributions-wikimedia</code>. After the component is available on <code>install*</code> servers, you can upload to it by running:
To create a new component, you simply need to prepare a patch for <code>puppet/modules/aptrepo/files/distributions-wikimedia</code>. After the component is available on <code>install*</code> servers, you can upload to it by running:
   '''$ reprepro -C component/thumbor include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes'''
   $ reprepro -C component/thumbor include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes
Note: if you get an error like <code>Error opening config file './conf/distributions': No such file or directory(2)</code>, then you forgot to do <code>sudo -H bash</code>


== Create a Debian patch ==
== Create a Debian patch ==
Line 101: Line 105:


1. Check if your source package is using quilt:
1. Check if your source package is using quilt:
   '''$ cat debian/source/format'''
   $ cat debian/source/format
   3.0 (quilt)
   3.0 (quilt)


2. Make the edits you wish to do for your fix and run the following command:
2. Make the edits you wish to do for your fix and run the following command:
  '''$ dpkg-source --commit'''  
  $ dpkg-source --commit   
  librsvg-2.40.18/tests/crash.c
  librsvg-2.40.18/tests/crash.c
  Enter the desired patch name: something-with-sense.patch
  Enter the desired patch name: something-with-sense.patch
Line 121: Line 125:


4. Your patch will be saved under <code>debian/patches/foo.patch</code>. Build it with the usual way:
4. Your patch will be saved under <code>debian/patches/foo.patch</code>. Build it with the usual way:
  '''$ DIST=stretch pdebuild'''   -> (use packages from stretch repo)
  $ DIST=stretch pdebuild  -> (use packages from stretch repo)
  or  
  or  
  '''$ DIST=stretch-wikimedia pdebuild''' -> (include packages from stretch-wikimedia repo)
  $ DIST=stretch-wikimedia pdebuild  -> (include packages from stretch-wikimedia repo)


== Misc Notes ==
If you are uploading a package to e.g. <code>buster-wikimedia</code> that is a patched version of the version originally in Debian proper, you should also pass <code>--debbuildopts -sa</code> to pdebuild.  This will force it to include the <code>.orig.tar.gz</code> as well.
* If we are building a new package where we are the owners, like python-thumbor-wikimedia, we should use <code>native</code> instead of <code>quilt</code> in <code>debian/source/format</code>
 
* If we are to use packages from eg php72 component, this can be done by using PHP72=yes env var before building
== Browse current package source ==
To review the active patches we have applied to a package, there are two approaches one can take: standalone using <code>dget</code>, or through <code>apt source</code>. The former has the benefit of not needing to modify any <code>/etc/apt/</code> configuration in the environment where you do this. The latter has the benefit of using the same tooling and logic as installing a package would (and thus involve changing /etc/ files).
 
=== Standalone using <code>dget</code> ===
 
* Find the latest <code>dsc</code> file for the package from https://apt.wikimedia.org/wikimedia/pool/. For example, kafkacat from the main component is under https://apt.wikimedia.org/wikimedia/pool/main/k/kafkacat/. Our custom component/php72 packages are under https://apt.wikimedia.org/wikimedia/pool/component/php72/p/php7.2/
* Ensure [https://packages.debian.org/buster/devscripts devscripts] is installed, which provides the <code>dget</code> command.
* Move to a temporary directory and run <code>dget -u <nowiki>https://apt.wikimedia.org/wikimedia/.../example-1.2.3-wmf1.dsc</nowiki></code>
* The package source is now in <code>example-1.2.3-wmf1/</code>, with a familiar "debian" subdirectory containing "debian/patches", etc.
 
=== Through apt ===
 
* Ensure <code>/etc/apt/sources.list</code> also includes the deb-src variant of the relevant sources.list lines. For example, when using component/php72 add the last line as well:
 
deb <nowiki>http://mirrors.wikimedia.org/debian/</nowiki> buster main
deb <nowiki>http://security.debian.org/</nowiki> buster/updates main
deb <nowiki>http://mirrors.wikimedia.org/debian/</nowiki> buster-updates main
deb <nowiki>http://mirrors.wikimedia.org/debian/</nowiki> buster-backports main contrib
deb <nowiki>http://apt.wikimedia.org/wikimedia</nowiki> buster-wikimedia main
deb <nowiki>http://apt.wikimedia.org/wikimedia</nowiki> buster-wikimedia component/php72
deb-src <nowiki>http://apt.wikimedia.org/wikimedia</nowiki> buster-wikimedia component/php72
 
* Move to a temporary directory and run <code>apt source <pgk-name></code>, e.g. <code>apt source php7.2</code>. This will use the same resolution logic as for <code>apt-get install</code>.
* The package source is now in <code><pgk-name>-1.2.3-wmf1/</code>, with a familiar "debian" subdirectory containing "debian/patches", etc.
 
=== See also ===
If a package is unchanged from upstream, then you can also browse the source online via Debian Salsa and its GitLab interface. For example:
 
https://salsa.debian.org/php-team/php/-/tree/debian/main/7.2/debian/patches
 
== See also ==
 
* [[Debmonitor]]
* [[Debian]]


[[Category:Package management|*]]
[[Category:Package management|*]]
[[Category:Runbooks]]
[[Category:SRE Infrastructure Foundations]]

Revision as of 14:31, 12 August 2022

These are runbooks related to Debian packaging for Wikimedia Foundation production.

Rebuilding a package

Process to simple repackage a debian package and push it to wmf apt repo.

Example: repackage python-libthumbor for debian stretch

1. Login to build2001.codfw.wmnet:

2. As a user run the following to download python-libthumbor's source package: 

$ apt-get source libthumbor=1.3.2-0+wmf1

3. We should now have 3 files and 1 directory in our working directory: 

 -rw-r--r--  1 user wikidev  15K Jul  6  2016 libthumbor_1.3.2.orig.tar.gz          -> source code of the upstream authors
 -rw-r--r--  1 user wikidev 1.2K Jul  6  2016 libthumbor_1.3.2-0+wmf1.dsc           -> meta file describing which files are around
 -rw-r--r--  1 user wikidev 2.4K Jul  6  2016 libthumbor_1.3.2-0+wmf1.debian.tar.xz -> Debian-specific changes
 drwxr-xr-x  6 user wikidev 4.0K Nov 20 14:11 libthumbor-1.3.2                      -> extracted source package

4. In the libthumbor-1.3.2/debian directory, we may find the following files: 

 changelog -> records all changes and defines the version of the package
 compat -> a meta file used for automation of the package build
 control -> defines the binary package names and their relation to other packages (e.g. files needed to build and run the package)
 copyright -> decribes the license and the copyright holders
 docs -> this can be used to declare some files as documentation
 patches -> a directory of code changes done to the original tarball
 rules -> defines the package build process
 source -> a directory defining some details of how the source package is defined
 watch -> a meta file which contains information where to look for new releases (i.e. where to download a newer source)

5. Go to libthumbor-1.3.2: 

$ libthumbor-1.3.2
$ dch -i

6. We should see something like the following: 

 libthumbor (1.3.2-0+wmf1+stretch1) stretch-wikimedia; urgency=low
 
  * Rebuild for stretch
 
 -- Effie Mouzeli <effie@wikimedia.org>  Tue, 20 Nov 2018 13:42:24 +0000
 
 libthumbor (1.3.2-0+wmf1) unstable; urgency=low
 
  * Initial release. (Closes: #783577)
 
 -- Gilles Dubuc <gilles@wikimedia.org>  Thu, 19 May 2016 15:30:16 +0000

7. Add your new version string, make sure it is a higher version than the previous one.

If rebuilding for a newer Debian version (eg. Buster)

Do: dch -bv 0.1.7-1~buster0 (that's the current version, plus "~buster0")

when we rebuild for a new distro we use ~, not + like in the example in that page, the difference being 1.5-1~something is less than 1.5-1 so if 1.5-1 ever appears on official buster, it will supersede our package.

And change UNRELEASED to buster-wikimedia

8. Build! 

$ DIST=stretch pdebuild   -> (use packages from stretch repo)
or 
$ DIST=stretch-wikimedia pdebuild  -> (include packages from stretch-wikimedia repo)

If building for a new Debian version do something like: DIST=buster WIKIMEDIA=yes pdebuild --debbuildopts -sa

9. Find the build results: 

$ ls  -la /var/cache/pbuilder/result/stretch-amd64/*thum*
-rw-r--r-- 1 user wikidev 5873 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/libthumbor_1.3.2-0+wmf1+stretch1_amd64.buildinfo
-rw-r--r-- 1 user wikidev 1797 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/libthumbor_1.3.2-0+wmf1+stretch1_amd64.changes
-rw-r--r-- 1 user wikidev 2500 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/libthumbor_1.3.2-0+wmf1+stretch1.debian.tar.xz
-rw-r--r-- 1 user wikidev 1260 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/libthumbor_1.3.2-0+wmf1+stretch1.dsc
-rw-r--r-- 1 user wikidev 7862 Nov 20 14:12 /var/cache/pbuilder/result/stretch-amd64/python-libthumbor_1.3.2-0+wmf1+stretch1_all.deb

Download source packages from snapshot.debian.org

If you are looking to build a different version than the one provided by a source package, you can have a look at Debian snapshots. Everything you need will be downloaded for you by pointing dget to the relevant .dsc file: 

$ dget https://snapshot.debian.org/archive/debian-debug/20180826T210249Z/pool/main/libr/librsvg/librsvg_2.40.20-3.dsc

Common problems

Sometimes the source package you are backporting depends upon a newer version of debhelper than is available in the target distribution: 

 pbuilder-satisfydepends-dummy : Depends: debhelper (>= 9.20160709) but 9.20150101+deb8u2 is to be installed.

Adding BACKPORTS=yes to the beginning of your pdebuild invocation will often fix this.

Misc notes

  • If we are building a new package where we are the owners, like python-thumbor-wikimedia, we should use native instead of quilt in debian/source/format
  • If we are to use packages from eg php72 component, this can be done by using PHP72=yes env var before building.

Upload to Wikimedia Repo

1. Login to apt1001.wikimedia.org, create a directory and copy all files from build2001 

$ ssh apt1001.wikimedia.org
$ mkdir libthumbor; cd libthumbor
$ rsync rsync://build2001.codfw.wmnet/pbuilder-result/stretch-amd64/*thum* .

2. Import! 

$ sudo -H bash
$ reprepro -C main include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes

Note: We split the archives sometimes, i.e. we add packages to components instead of main

3. Remember to log your uploads in #wikimedia-operations on IRC using !log

4. Achievement Unlocked!

Upload to a component directory

Sometimes we build packages for specific purposes thus we don't want them to be available under the main component. To create a new component, you simply need to prepare a patch for puppet/modules/aptrepo/files/distributions-wikimedia. After the component is available on install* servers, you can upload to it by running: 

 $ reprepro -C component/thumbor include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes

Note: if you get an error like Error opening config file './conf/distributions': No such file or directory(2), then you forgot to do sudo -H bash

Create a Debian patch

Sometimes we might wish to add a fix to the software we are packaging, not included in the original source.

1. Check if your source package is using quilt: 

 $ cat debian/source/format
 3.0 (quilt)

2. Make the edits you wish to do for your fix and run the following command: 

$ dpkg-source --commit  
librsvg-2.40.18/tests/crash.c
Enter the desired patch name: something-with-sense.patch

3. Name your patch and use the spawned editor to make a description, for instance: 

Description: add -exact in webp conversion
The -exact option was introduced in webp 0.5, which is only available on Stretch.

python-thumbor-wikimedia (2.2-1+deb9u1) stretch-wikimedia; urgency=low

  * Repackaging for stretch

Author: Effie Mouzeli <effie@wikimedia.org>
 ---

4. Your patch will be saved under debian/patches/foo.patch. Build it with the usual way: 

$ DIST=stretch pdebuild   -> (use packages from stretch repo)
or 
$ DIST=stretch-wikimedia pdebuild  -> (include packages from stretch-wikimedia repo)

If you are uploading a package to e.g. buster-wikimedia that is a patched version of the version originally in Debian proper, you should also pass --debbuildopts -sa to pdebuild. This will force it to include the .orig.tar.gz as well.

Browse current package source

To review the active patches we have applied to a package, there are two approaches one can take: standalone using dget, or through apt source. The former has the benefit of not needing to modify any /etc/apt/ configuration in the environment where you do this. The latter has the benefit of using the same tooling and logic as installing a package would (and thus involve changing /etc/ files).

Standalone using dget

Through apt

  • Ensure /etc/apt/sources.list also includes the deb-src variant of the relevant sources.list lines. For example, when using component/php72 add the last line as well:
deb http://mirrors.wikimedia.org/debian/ buster main
deb http://security.debian.org/ buster/updates main
deb http://mirrors.wikimedia.org/debian/ buster-updates main
deb http://mirrors.wikimedia.org/debian/ buster-backports main contrib
deb http://apt.wikimedia.org/wikimedia buster-wikimedia main
deb http://apt.wikimedia.org/wikimedia buster-wikimedia component/php72

deb-src http://apt.wikimedia.org/wikimedia buster-wikimedia component/php72
  • Move to a temporary directory and run apt source <pgk-name>, e.g. apt source php7.2. This will use the same resolution logic as for apt-get install.
  • The package source is now in <pgk-name>-1.2.3-wmf1/, with a familiar "debian" subdirectory containing "debian/patches", etc.

See also

If a package is unchanged from upstream, then you can also browse the source online via Debian Salsa and its GitLab interface. For example:

https://salsa.debian.org/php-team/php/-/tree/debian/main/7.2/debian/patches

See also

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Debian_Packaging&oldid=592713"