ReleaseEngineering/PuppetAgain/Packages
Principles
PuppetAgain needs to be able to repeatably produce identical machines over long time ranges (years). That means carefully controlling which versions of which packages are available to be installed, both during machine creation and in runs of the puppet agent. However, it's impractical and unnecessary to specify the exact version of *every* package installed on a host.
The system needs to be able to manage periodic updates to packages, both for security fixes and normal software upgrades. This must be done in a controlled fashion, so that any effects on system behavior can be identified and handled appropriately.
We often need updated versions of distro-supplied packages, custom packages for existing applications, or packaged versions of internally-developed tools.
How all of this happens differs widely among supported operating systems!
CentOS
CentOS: Repositories
Every puppetmaster hosts a bunch of yum repositories:
Notes:
- Paths under yum/releng are custom packages, and are not mirrored from anywhere.
- We generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind.
- The Fedora repos are only used within mock on buildslaves. They are not used to install anything on puppet-managed hosts.
- The releng Fedora repos are currently empty, in fact
- Dynamic repositories are snapshots that are made on demand, where latest always points to the latest active snapshot. They are *not* automatically updated. The date on which they were most recently mirrored is given above.
- The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
- The CentOS base repos each have an images/ subdirectory, so they can be used with the 'url' command in a kickstart script
Mirror Synchronization Commands
Note: Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See bug 836014. The process takes about 10 minutes.
Note: From inside Mozilla, you'll need to use the Squid proxy: set RSYNC_PROXY to proxy.dmz.scl3.mozilla.com:3128
Remember that once changes land on the puppet master, they can be used for any purpose that needs packages. That includes kickstart, which always takes the most recent version of a package it can find! It also includes installs by puppet that specify version => "latest", and installs to fulfill requirements of packages defined by puppet. Think twice!
EPEL
rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/x86_64/ /data/repos/yum/mirrors/epel/6/2012-03-07/x86_64/ rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/i386/ /data/repos/yum/mirrors/epel/6/2012-03-07/i386/ time hardlink -v /data/repos/yum/mirrors/epel
Note that, as for CentOS below, the i386 mirrors were done much later than 2012-03-07 (July 2012).
CentOS 6
- Base and Updates (6.3 and higher)
CENTOS_FULL=6.5
rsync -v -n -aP --filter='-r centos-version.txt' --exclude isos --exclude drpms --exclude centosplus --exclude xen4 --exclude fasttrack --exclude contrib --exclude cr \
--delete --delete-excluded rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_FULL/
time hardlink -v /data/repos/yum/mirrors/centos
Fedora 16
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/fedora/linux/releases/16/Everything/i386/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/i386/os/ rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://fedora.osuosl.org/fedora/linux/releases/16/Everything/x86_64/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/x86_64/os/ rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/i386/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/i386/ rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/x86_64/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/x86_64/ time hardlink -v /data/repos/yum/mirrors/fedora/
PuppetLabs
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64/ rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/products/i386/ rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/dependencies/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64/ rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/dependencies/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/dependencies/i386/ time hardlink -v /data/repos/yum/mirrors/puppetlabs/
Passenger
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/x86_64/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/x86_64/ rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/ time hardlink -v /data/repos/yum/mirrors/passenger/
HP
not everything is mirrored, so this just gets the good stuff
rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/x86_64/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/ rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/i386/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/ time hardlink -v /data/repos/yum/mirrors/hp/
CentOS: Installing Packages with Puppet
CentOS Packages are simple to install:
CentOS: {
package {
"mypackage":
ensure => "1.2.3";
}
}
Generally, if it's important enough to install explicitly, it's important enough to pin a particular version. If you also need to pin versions for requirements, be sure you model the requirements with requires in puppet, so that puppet knows to install the requirements first.
CentOS: Custom Packages
See ReleaseEngineering/PuppetAgain/HowTo/Build RPMs for building RPMs.
CentOS: Updated Versions of Upstream Packages
In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (yum install http://wherever.it.is/package.rpm) on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories. If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems. Upgrading librsync may be OK, but upgrading glibc or libopenssl this way might lead to a world of pain and sadness (noting that security releases often don't!).
Before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run createrepo (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.
Note: Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream
CentOS: Landing Custom Repository Changes
Custom-built packages should be placed in the appropriate repos/yum/releng/public/CentOS/6/* repository, depending on architecture. All dependencies should be included in that repository if they are not in the mirrored repositories.
When the patch containing the new or updated package spec is r+'d, commit it as usual, and also add *both* the RPM (or multiple RPMs if multiple architectures are required!) and the SRPM into /data on the designated puppet master, so that it will be distributed to other systems. Debuginfo RPMs are a good idea, too.
Run
createrepo --update $repo_path
to update the metadata. Don't forget to run puppetmaster-fixperms to make sure permissions are correct.
Once this is done, the package is available and will be used if possible.
Example session:
# at releng-puppet2.srv.releng.scl3.mozilla.com $ wget http://people.mozilla.org/~jhopkins/bug772446/supervisor-3.0-0.10.a12.el6.noarch.rpm $ wget http://people.mozilla.org/~jhopkins/bug772446/supervisor-3.0-0.10.a12.el6.src.rpm $ chmod 644 *.rpm $ sudo chown puppetsync: *.rpm $ sudo mv -vi *.rpm /data/repos/yum/releng/public/CentOS/6/noarch/ `supervisor-3.0-0.10.a12.el6.noarch.rpm' -> `/data/repos/yum/releng/public/CentOS/6/noarch/supervisor-3.0-0.10.a12.el6.noarch.rpm' `supervisor-3.0-0.10.a12.el6.src.rpm' -> `/data/repos/yum/releng/public/CentOS/6/noarch/supervisor-3.0-0.10.a12.el6.src.rpm' $ sudo createrepo --update /data/repos/yum/releng/public/CentOS/6/noarch 2/2 - supervisor-3.0-0.10.a12.el6.src.rpm Saving Primary metadata Saving file lists metadata Saving other metadata # set permissions and ownership on repodata and package files $ sudo puppetmaster-fixperms
Ubuntu
Reference Links
Ubuntu: Repositories
| url | repository | arch | section | dist | mirror date |
|---|---|---|---|---|---|
| repos/apt/ubuntu | Ubuntu 12.04 LTS | i386,amd64 | main,restricted,universe | precise,precise-security (note: no precise-updates) | 2013-02-21 |
| repos/apt/xorg-edgers | xorg-edgers fresh X Crack | i386,amd64 | main,restricted,universe | precise | 2013-02-21 |
| repos/apt/releng | custom-built packages | i386,amd64 | main,restricted,universe | precise | |
| repos/apt/releng-updates | partial mirror of precise-updates | i386,amd64 | main,restricted,universe | precise | |
| repos/apt/puppetlabs | mirror of apt.puppetlabs.com | i386,amd64 | main,restricted,universe | precise |
Notes:
- Dynamic repositories are snapshots that are made on demand, where latest always points to the latest active snapshot. They are *not* automatically updated. The date on which they were most recently mirrored is given above.
- This arrangement can't be updated (see the warning below), so we will likely need to change to something more like the CentOS repositories.
Mirror Synchronization Commands
Warning: The debmirror command includes the equivalent of rsync's --delete. If you try to mirror only one section, dist, arch, etc., debmirror will delete the section/dist/arch you omit. Don't do that. See bug 844172.Setup
GNUPGHOME has the Ubuntu arch key in it. If a sync operation fails because a signature does not verify, download the key using
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys $KEY_ID
Add `--keyserver-options http-proxy=proxy.dmz.scl3.mozilla.com:3128` at Mozilla. Note that this keyserver's search option appears to be broken. You can usually google for the key id, and find the relevant link on the keyserver, and then copy-paste the result into
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import
with `/etc/debmirror.conf` containing only the Perl no-op "1;".
Note that `hardlink` isn't used here - debmirror's name-based deduplication squeezes out just about all the space possible.
Ubuntu
We mirror all Ubuntu releases to the same directory, using --nocleanup to prevent deletion of packages not touched in the current mirror operation. Note that we mirror 'universe', too. Although it's huge, puppet relies on some packages in that section.
For example, the following mirrors precise. The DIST can be changed to e.g., mirror a different version, or just mirror security.
SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer DIST=precise,precise-security ARCH=i386,amd64 GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \ -a $ARCH -s $SECTION -d $DIST \ -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \ --nocleanup \ --dry-run \ /data/repos/apt/ubuntu/
xorg-edgers
SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer DIST=precise ARCH=i386,amd64 GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \ -a $ARCH -s $SECTION -d $DIST \ -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \ --dry-run \ --nocleanup \ /data/repos/apt/xorg-edgers/
puppetlabs
wget https://apt.puppetlabs.com/pubkey.gpg GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import pubkey.gpg
SECTION=main,dependencies
DIST=precise,trusty
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
-a $ARCH -s $SECTION -d $DIST \
-h apt.puppetlabs.com -r / -e http --progress \
--dry-run \
/data/repos/apt/puppetlabs/
Ubuntu: Installing Packages with Puppet
Simple:
Ubuntu: {
package {
"mypackage":
ensure => "1.2.3";
}
}
As with CentOS packages, it's generally a good idea to pin the version of things that are important enough to be named in puppet. If requirements need to be pinned too, then model the requirements relationship properly with requires.
Ubuntu: Custom Packages
See ReleaseEngineering/PuppetAgain/HowTo/Build DEBs for details on building DEBs.
Ubuntu: Updated Versions of Upstream Packages
Warning: This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the releng-updates repository) and packages installed by puppet (which does).Sometimes you may need to update only one package from upstream without syncing the whole repo (what may cause unpredictable results). The releng-updates repo is set up for these cases.
Adding a Single Package
To do this, you'll need apt-ftparchive from the apt-utils package. It only runs on Ubuntu, so rsync the releng-updates repo to an Ubuntu host and rsync the results back when you're done.
Put the new packages in the appropriate place under releng-updates/pool. This might be easiest by rsyncing from archive.ubuntu.org, e.g.,
rsync -r rsync://archive.ubuntu.com/ubuntu/pool/main/o/openssl/ releng-updates/pool/main/o/openssl/
(after which I deleted the files I didn't need).
Once the pool is set up, regenerate the packages with the following:
rm -rf releng-updates/dists
cd releng-updates
for arch in i386 amd64; do
for dist in precise trusty; do
mkdir -p dists/${dist}-updates/all/binary-$arch
apt-ftparchive --arch $arch packages . > dists/${dist}-updates/all/binary-$arch/Packages
bzip2 < dists/${dist}-updates/all/binary-$arch/Packages > dists/${dist}-updates/all/binary-$arch/Packages.bz2
done
done
Automatically Pulling Dependencies
This needs to be done on an Ubuntu machine. You need to use debpartial-mirror, apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes:
# http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates.sh debpartial-mirror -c ./releng-updates.conf all rm -rf releng-updates/dists cd releng-updates for arch in i386 amd64; do mkdir -p dists/precise-updates/all/binary-$arch apt-ftparchive --arch $arch packages . > dists/precise-updates/all/binary-$arch/Packages bzip2 < dists/precise-updates/all/binary-$arch/Packages > dists/precise-updates/all/binary-$arch/Packages.bz2 done
[GLOBAL] ;debug = DEBUG mirror_dir = ./ architectures = i386 amd64 components = main distributions = precise get_suggests = true get_recommends = true get_provides = true get_sources = true get_packages = true [releng-updates] server = http://archive.ubuntu.com/ubuntu distributions = precise-updates components = main restricted universe # specify needed packages here filter = name:gnome-settings-daemon
Ubuntu: Landing Custom Repository Changes
Note: This is currently Mozilla-only, since Mozilla is the upstream for all packages. Be careful, as this can easily wipe out the existing releng repository!
Warning: Debian repositories can only contain one version of a package. When you land an upgrade into a repo, land the corresponding puppet change that will upgrade the package as soon as possible. Remember to bump the repo version (below).Note: these steps were used for libxcb1 and Mesa.
We use reprepro to manage packages. Nobody's quite sure how to pronounce it.
The repo lives in /data/repos/apt on releng-puppet2.srv.releng.scl3, so you'll need to begin by rsync'ing it to ubuntu64packager1:
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/ /data/repos/apt/releng/ rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/conf/ /data/repos/apt/conf/ rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/ /data/repos/apt/db/
The conf/ directory contains config files pointing to the releng/ repository only -- this won't (shouldn't!) mess with the Ubuntu mirrors.
Next, import your files into the repo
cd /data/repos/apt reprepro -V --basedir . include precise /tmp/somepackage-precise-amd64/*.changes
NOTE: You will have to place a copy of the orig.tar.gz in the repo.
If you're installing two architectures, then the second one will complain that the debian tarball's signature doesn't match. That's OK. Just import the debs from that architecture:
reprepro -V --basedir . includedeb precise /tmp/somepackage-precise-i386/*.deb
This will copy packages to releng/, generate indices and update the database. Make sure to not get into a race condition with other people on the same host.
When you are done with the repo, first check that any previous versions of the packages you're updating are pinned in puppet. Then rsync the repo and db back to the distinguished master:
rsync -av /data/repos/apt/releng/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/ rsync -av /data/repos/apt/db/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/
and run this on the master:
puppetmaster-fixperms
You need to bump the counter in modules/packages/manifests/setup.pp:
# to flush the package index, increase this value by one (or # anything, really, just change it). - $repoflag = 5 + $repoflag = 6
This will cause all Ubuntu machines to run apt-get update.
Darwin
Darwin: Repositories
There's no such thing as a repository for OS X packages, sadly. DMGs are stored in repos/DMGs. DMGs are generally built for a specific OS version and put in per-os-version subdirectories, although DMGs that are compatible across versions are in the root. Each DMG is named $packagename-$version.dmg.
Darwin: Installing Packages with Puppet
Use the packages::pkgdmg defined type to install DMGs, giving the package name as the resource name and the package version in the version parameter. The type will construct the correct filename from this information. For example:
Darwin: {
Anchor['packages::wget::begin'] ->
packages::pkgdmg {
wget:
version => "1.12-1";
} -> Anchor['packages::wget::end']
}
DMGs that are downloaded as-is should be documented briefly (at least with the source) in the manifest files that install them. For example (from modules/packages/manifests/puppet.pp):
Darwin: {
# These DMGs come directly from http://downloads.puppetlabs.com/mac/
Anchor['packages::puppet::begin'] ->
packages::pkgdmg {
puppet:
version => $puppet_version;
facter:
version => $facter_version;
} -> Anchor['packages::puppet::end']
}
For DMGs that are *not* os-version-specific, pass os_version_specific => false.
Darwin: Custom Packages
DMGs that are custom built should have a shell script in modules/packages/manifests named $package-dmg.sh which builds the DMG. If there is a corresponding RPM (custom or stock) for the package, then the shell script can require that the source RPM be unpacked first. See ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs for more details.