ReleaseEngineering/PuppetAgain/HowTo/Build DEBs

Importing particular updates from upstream

Sometimes you may need to update only one package from upstream without syncing the whole repo (what may cause unpredictable results). releng-updates repo is set up for these cases.

You need to use debpartial-mirror, apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes:

# http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates.sh
debpartial-mirror -c ./releng-updates.conf all
rm -rf releng-updates/dists
cd releng-updates
for arch in i386 amd64; do
  mkdir -p dists/precise-updates/all/binary-$arch
  apt-ftparchive --arch $arch packages . > dists/precise-updates/all/binary-$arch/Packages
  bzip2 < dists/precise-updates/all/binary-$arch/Packages > dists/precise-updates/all/binary-$arch/Packages.bz2
done

releng-updates.conf

[GLOBAL]
;debug = DEBUG
mirror_dir = ./
architectures = i386 amd64
components = main
distributions = precise

get_suggests = true
get_recommends = true
get_provides = true
get_sources = true
get_packages = true

[releng-updates]
server = http://archive.ubuntu.com/ubuntu
distributions = precise-updates
components = main restricted universe
# specify needed packages here
filter = name:gnome-settings-daemon

Building

At the moment we don't have any shared resources to build Debian packages. You need to use Ubuntu 12.04 based VM on your laptop.

To build a package in clean-room environment use pbuilder. It uses precreated images to build DEBs in chroot.

Create pbuilder images

Use the following command to create environments.

• 64 bit

sudo pbuilder --create --distribution precise --architecture amd64 \
    --components "main restricted universe multiverse" \
    --debootstrapopts --variant=buildd \
    --basetgz /var/cache/pbuilder/base-precise-amd64.tgz \
    --mirror http://archive.ubuntu.com/ubuntu \
    --othermirror "deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse" \
    --othermirror "deb http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse"

• 32 bit

sudo pbuilder --create --distribution precise --architecture i386 \
    --components "main restricted universe multiverse" \
    --debootstrapopts --variant=buildd \
    --basetgz /var/cache/pbuilder/base-precise-i386.tgz \
    --mirror http://archive.ubuntu.com/ubuntu \
    --othermirror "deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse" \
    --othermirror "deb http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse"

Building a package

There are some different ways to build a binary package

Building from orig.tar.gz, debian.tar.gz and dsc

sudo pbuilder --build --distribution precise --architecture amd64 --basetgz /var/cache/pbuilder/base-precise-amd64.tgz\
    --buildresult ./out *.dsc

"out" directory will be populated with files you'll need to import into the repo.

By default if the same upstream version has 2 entries in debian/changelog, packaging tools don't add orig.tar.gz in the *.changes files (used when you import the package). If for some reason you need to add the upstream source into the "changes" file (for example, when you backport some package) pass "-sa" to the packaging tools:

sudo pbuilder --build --distribution precise --architecture amd64 --basetgz /var/cache/pbuilder/base-precise-amd64.tgz \
   --buildresult out --debbuildopts "-sa" *.dsc

Building using git-buildpackage

git-buildpackage allows you to have everything related to a package under version control. Your git checkout contains the following branches needed to build the package:

• upstream: upstream sources
• debian: debianization work
• pristine-tar (optional): binary information needed to recreate orig.tar.gz from git with a proper final checksum

The following config (~/.gbp.conf) file is used:

[DEFAULT]
cleaner = fakeroot debian/rules clean
pristine-tar = True
color = auto

[git-buildpackage]
export-dir = ../build-area/
tarball-dir = ../tarballs/
pbuilder = True
dist = precise

[git-import-orig]
dch = False

The following scenario used to rebuild puppet-2.7.17:

# get Debian's package (apt-cache showsrc puppet|grep ^Vcs-Git:)
git clone git://git.debian.org/git/pkg-puppet/puppet.git
cd puppet
git co -b pristine-tar origin/pristine-tar
git co -b mozilla upstream/2.7.17 # checkout upstream version we need, by tag
git merge debian/2.7.17-1 # merge debian changes for that version, by tag
dch --local mozilla --distribution precise # bump debian/changelog with proper version suffix
git commit -va
# biuld it
ARCH=amd64 BUILDER=pbuilder git-buildpackage --git-upstream-branch=origin/upstream --git-debian-branch=mozilla  # add -sa to include source)
# once you're happy with the package, tag it
ARCH=amd64 BUILDER=pbuilder git-buildpackage --git-upstream-branch=origin/upstream --git-debian-branch=mozilla --git-tag
# to build i386 specific package you need to pass -B to avoid adding orig/debian.tar/gz files into the changes file
ARCH=i386 BUILDER=pbuilder git-buildpackage --git-upstream-branch=origin/upstream --git-debian-branch=mozilla  -B

Building multiple dependent packages

Sometimes you may need to build a package which requires a fresher dependency (library). In this case you need to build the library first and use a temporary local repo. Packaging nodejs is a good example.

Pbuilder can use hooks to run some actions.

• Add the following entries to your ~/.pbuilderrc

HOOKDIR="$HOME/.pbuilder-hooks"
OTHERMIRROR="deb file://$HOME/debs/packages ./"
BINDMOUNTS="$HOME/debs/packages"
EXTRAPACKAGES="apt-utils"

• Add a hook to generate package indexes

mkdir ~/.pbuilder-hooks

• .pbuilder-hooks/D05deps contents:

#!/bin/sh
(cd /home/rail/debs/packages && apt-ftparchive packages . > Packages)
apt-get update

• chmod 755 .pbuilder-hooks/D05deps

• edit existing images

sudo pbuilder --login --basetgz /var/cache/pbuilder/base-precise-amd64.tgz --save-after-login
# inside the subshell
apt-get install apt-utils
echo "deb file:///home/rail/debs/packages ./" >> /etc/apt/sources.list
echo 'APT::Get::AllowUnauthenticated "true";' > /etc/apt/apt.conf.d/90mozilla

• repeat for the i386 image

• build the library
• copy the resulting debs into $HOME/debs/packages
• build the application

Testing

At the moment there is no testing repo. You need to manually install packages to test them.

Landing

NOTE: be careful with this not-yet-ready-to-production version.

We use reprepro to manage packages.

ATM, the repo lives in /data/repos/apt on releng-puppet1.srv.releng.scl3.

There are 2 important files under conf directory:

• options

verbose
outdir +b/releng
logdir +b/logs
dbdir +b/db
morguedir +b/morgue
keepunreferencedfiles
keepunusednewfiles

• distributions

Origin: mozilla
Label: Mozilla
Codename: precise
Version: 12.04
Architectures: amd64 i386 source
Components: main
Description: Releng repos
DebIndices: Packages Release . .bz2

Until we have a shared location (an Ubuntu machine) for package building and repo updates (using incoming directory) you need to sync that directory to your machine.

Example import:

cd repo
reprepro -V --basedir . include precise ~/debs/puppet/build-area/puppet_2.7.17-1mozilla1_amd64.changes

It will copy packages to releng/, generate indices and update the database. Make sure to not get into a race condition with other people.

When you are done with the repo rsync it back, then publish:

sudo chown puppetagainsync:puppetagainsync -R .
sudo rsync -av ./ /data/repos/apt/