You are browsing a read-only backup copy of Wikitech. The primary site can be found at


From Wikitech-static
Jump to navigation Jump to search


Notes on building mwlib 'offline collection generator' hosts . . .

ocg1 installed - testing cherrypicked-to-production puppet work here
ocg2 installed and are awaiting puppetization
ocg3 has a firmware issue that needs resolution


  • use site-packages?
  • use dh_python2?

Should these be isolated from stock python packages, i.e. in an alternate library path, etc?
dpkg-buildpackage -rfakeroot -b

Labs pediapress project

bob - build machine, ip
pediapress-ocg1 - ubuntu precise test host
pediapress-ocg2 - ubuntu lucid test host
pediapress-packager - ubuntu lucid host for packaging

To Be Determined

1) daemonizing: runit, init, upstart?

2) package individual modules or entire homedir heap as a single package?

3) depending on #2, install modules system-wide or under a user?

4) other user and filesystem layout specifics not covered in 1-3

Build process

puppet does this already

 apt-get install -y dvipng g++ gcc git-core imagemagick libfreetype6-dev libjpeg-dev liblcms-dev \
    libxml2-dev libxslt-dev libz-dev make ocaml-nox pdftk ploticus python python-dev python-docutils \
    python-greenlet python-imaging python-lxml python-pyparsing python-pypdf python-virtualenv \
 groupadd pp
 useradd -m -c "pediapress" -g pp -d /opt/pp pp

this part needs puppetification

 su - pp
 virtualenv --distribute ~/local; export PATH=~/local/bin:$PATH; hash -r; export PIP_INDEX_URL=
 pip install pil       # provided by python-imaging
 pip install pyfribidi # no additional dependencies
 pip install mwlib     # pulls odfpy-0.9.4, gevent-1.0b2, timelib-0.2.3, bottle-0.10.9, apipkg-1.0, qserve-0.2.7, roman-1.4.0, py-1.4.7, sqlite3dbm-0.1.4
 pip install mwlib.rl  # pulls mwlib.ext, pygments-1.5
 pip install mwlib.zim # pulls pyzim
 set up daemons

Packages to Build

Stock Packages That Should Work

 ubuntu: python-apipkg (lucid:n/a precise:1.0-1.1build1)
 ubuntu: python-bottle (lucid:0.6.4-1 precise:0.10.6-1)
 source package: git://
 stock package looks tricky to backport due to debian/control build dependencies
 ubuntu: python-pygments (lucid:1.2.2+dfsg-1ubuntu1 precise:1.4+dfsg-2)
 on precise, it's pulled in as a python-docutils dependency
 ubuntu: libc-ares-dev (lucid:1.7.0-1-1 precise:1.7.5-1)
 ubuntu: libc-ares2 (lucid:1.7.0-1 precise:1.7.5-1)
 ubuntu: libev-dev (lucid:1:3.8-1 precise:1:4.11-1)
 ubuntu: libev3 (lucid:1:3.8-1)
 ubuntu: libev4 (precise:1:4.11-1)
 ubuntu: python-reportlab (lucid:2.4-1 precise:2.5-1.1build1)
 ubuntu: python-reportlab-accel (lucid:2.4-1 precise:2.5-1.1build1)
 ubuntu: cython (lucid:0.11.2-1 precise:0.15.1-1ubuntu1)
 ubuntu: python-lxml (lucid:2.2.4-1 precise:2.3.2-1)
 ubuntu: libcxxtools-dev (lucid:1.4.8-3 precise:2.0-4ubuntu2)
 ubuntu: libtntdb-dev (lucid:1.0.1-3build1 precise:1.1-1)
 ubuntu: (?) xz-utils (lucid:4.999.9beta+20091116-1 precise:5.1.1alpha+20110809-3)

New Packages That Build Easily

 * built for lucid & precise
 * built for lucid & precise
 * built for lucid & precise
 * built for lucid & precise
 ubuntu: python-pyfribidi (lucid:0.10.0-2 precise:0.10.0-4), both patched for buffer overflow
 upstream appears dead, pediapress released 0.11.0 but changelog ends at 0.10.1
 comment: 0.11 and 0.10.1 have the same API. 0.10.0 won't work and is still broken on ubuntu precise
 notes: remove fribidi-src before packaging
 * built for lucid & precise but needs to either replace stock package or be packaged with a different name

New Packages That Require Significant Refactoring Before Build

pyzim (& zimlib, zimwriter)
 build requires: cxxtools =2.0, tntdb =1.1 (for zimwriter), tntnet =2.0 (for zimreader), libxz-dev, autoconf, automake, libtool
 uses a Makefile to build cxxtools, tntdb, xz, zimlib and zimwriter so you'll want remove the vendor directory
 - fulfill requires from stock cxxtools, tntdb, xz
 - build zimlib, pyzim, zimwriter as necessary as discrete packages
 backporting 1.0b2 to fix bugs Ralf patched and sent upstream in previous versions 
 ubuntu: python-gevent (lucid:n/a precise:0.13.6-1ubuntu1)
 stable: gevent-0.13.7
 beta: gevent-1.0b2 requires greenlet, libev & c-ares, release tarball includes dependencies
 significant changes between 0.13.7 and 1.0b2
 remove included libev and c-ares
 - fulfill requires from stock packages
 - create python-pediapress-gevent from 1.0b2 source

 "mwlib itself also includes a slightly patched apipkg (v1.1) module. I think I already
 got the changes merged upstream. mwlib also includes a patched script."
 - determine whether stock apipkg will suffice
 - otherwise backport as a replacement for stock mwlib  
 - possibly package separately
 provides external dependencies needed by the mwlib library
 essentially reportlab 2.5 with some patches (no Makefile)
 - determine whether stock reportlab will suffice
 - otherwise backport as a replacement for stock reportlab 
 check with Pediapress re. font install
 requires cython, lxml, mwlib, pyzim (pyzim requires zimlib, zimwriter) (no Makefile)
 - fulfill requirements with stock packages
 - build mwlib.zim itself as new package
 ./ext-date-lib is an import of php-5.2.10/ext/date/lib for use in static linking
 "timelib includes unmodified sources from php. you should also break those out
 and link against libphp (if there such a thing in ubuntu)."
 * first attempt package failed, won't build as fakeroot
 - preferred: reimplement in python
 - good: rewrite dynamically linked against stock php
 - acceptable: rewrite statically linked against stock php
 - pediapress way
   git clone
   cd texvc; make; make install PREFIX=~/local
 - mediawiki way (?)
   version #? use mediawiki version #? what?
   is debian/copyright sane?
   is version sane for pediapress folks?
 recommend: consult mw and pediapress devs for instructions on what to package