Releases/Firefox 3.1b1:BuildNotes

Build Engineers

bhearsum
release tracking bug

Signed-off Revision

http://hg.mozilla.org/mozilla-central/rev/197f83ad7678

Tags

Build 1:

Module	Tag	Changeset
mozilla-central		http://hg.mozilla.org/mozilla-central/rev/197f83ad7678

All l10n repositories listed in l10n-changesets were tagged with FIREFOX_3_1b1_RELEASE: http://hg.mozilla.org/build/buildbot-configs/file/FIREFOX_3_1b1_RELEASE/mozilla2/l10n-changesets

Build data

Type	Build ID	Build machine
[Windows installer/zip]	20081007144708	moz2-win32-slave11
[Mac compressed]	20081007125523	bm-xserve19
[Linux compressed]	20081007112159	moz2-linux-slave05

Notes

• Cleared up disk space before starting.
  • Removed old actionmonkey directories on linux slaves
• Free space before starting:
  • moz2-linux-slave01: 13G free
  • moz2-linux-slave02: 16G free
  • moz2-linux-slave03: 16G free
  • moz2-linux-slave05: 12G free
  • moz2-linux-slave06: 16G free
  • bm-xserve16: 25Gi free
  • bm-xserve17: 28Gi free
  • bm-xserve18: 25Gi free
  • bm-xserve19: 23Gi free
  • moz2-win32-slave01: 16G free
  • moz2-win32-slave02: 17G free
  • moz2-win32-slave03: 17G free
  • moz2-win32-slave05: 16.5G free
  • moz2-win32-slave06: 15.5G free
  • moz2-win32-slave11: 22G free
  • moz2-win32-slave12: 18G free
  • moz2-win32-slave13: 22G free

Build 1

Tag

• Tag failed out with a 'setupBuild' exception. Buildbot cannot handle Builders with more than 100 ShellCommands, see this line: http://mxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/process/base.py#397
  • Fixed by changing that line to '< 1000'
  • Upstream buildbot bug: http://buildbot.net/trac/ticket/366
  • Our bug: bug 458911
• Tag failed again because the 'hg.mozilla.org' host key wasn't accepted on the production slaves.
  • Went around and accepted the key on linux slaves 01, 02, 03, 05, 06, 11, 12 (mac and win32 machines do not push to hg)
• Tag failed again with a traceback when trying to push:

pushing to ssh://hg.mozilla.org/l10n-central/be
searching for changes
** unknown exception encountered, details follow
** report bug details to http://www.selenic.com/mercurial/bts
** or mercurial@selenic.com
** Mercurial Distributed SCM (version 0.9.5)
Traceback (most recent call last):
 File "/tools/python/bin/hg", line 14, in <module>
   mercurial.dispatch.run()
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 20, in run
   sys.exit(dispatch(sys.argv[1:]))
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 29, in dispatch
   return _runcatch(u, args)
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 45, in _runcatch
   return _dispatch(ui, args)
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 348, in _dispatch
   ret = _runcommand(ui, options, cmd, d)
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 401, in _runcommand
   return checkargs()
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 357, in checkargs
   return cmdfunc()
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 340, in <lambda>
   d = lambda: func(ui, repo, *args, **cmdoptions)
 File "/tools/python/lib/python2.5/site-packages/mercurial/commands.py", line 2147, in push
   r = repo.push(other, opts['force'], revs=revs)
 File "/tools/python/lib/python2.5/site-packages/mercurial/localrepo.py", line 1385, in push
   return self.push_unbundle(remote, force, revs)
 File "/tools/python/lib/python2.5/site-packages/mercurial/localrepo.py", line 1465, in push_unbundle
   return remote.unbundle(cg, remote_heads, 'push')
 File "/tools/python/lib/python2.5/site-packages/mercurial/sshrepo.py", line 195, in unbundle
   l = int(self.pipei.readline())
ValueError: invalid literal for int() with base 10: 
remote: abort: could not lock repository /repo/hg/mozilla/l10n-central/be:  Permission denied
program finished with exit code 1

• • Turns out that we never asked for 'ffxbld' to have push access to l10n repositories. Filed bug 458908 to get that enabled.

• Tagging went fine after this

• Two action items:
  • Dependent scheduler did not fire TODO bug 459213
  • Check-in comments say 'cltbld' when they should say 'ffxbld' TODO bug 459214

Source

• Automated, no problems.
• Manually pushed builds to stage:

# as cltbld@stage.mozilla.org
cd /home/ftp/pub/firefox/nightly
mkdir -p 3.1b1-candidates/build1
cd 3.1b1-candidates/build1
mkdir linux-i686 mac win32 update source
# as cltbld@moz2-linux-slave06 (where the source tarball/bundle were generated
cd /builds/moz2_slave/source
scp firefox-3.1b1* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/source

Build

• No problems with linux build
• Created *_info.txt files on stage
• Created en-US subdirs on stage:

cd /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1
mkdir linux-i686/en-US mac/en-US win32/en-US

• On moz2-linux-slave05:

cd /builds/moz2_slave/linux_build/build/obj-firefox
make package
make -C tools/update-packaging
# upload
scp firefox-3.1b1.en-US.linux-i686.tar.bz2 cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/linux-i686/en-US/firefox-3.1b1.tar.bz2
cd update
scp firefox-* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US/firefox-3.1b1.complete.mar

• Win32 and mac builds got interrupted when the Buildbot master hung because of graph server downtime. bug 458939
  • when the master came back up they were restarted
• A mac nightly build got uploaded by mistake (bug 458968), redid on bm-xserve19 with

# the old one moved to stage:~cltbld/wrong-mac-en-US-firefox-3.1b1.dmg
cd /builds/moz2_slave/macosx_build/build/obj-firefox/ppc
make package
make -C tools/update-packaging
scp dist/firefox-3.1b1.en-US.mac.dmg cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/mac/en-US/firefox-3.1b1.dmg
cd update
scp firefox-* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/update/mac/en-US/firefox-3.1b1.complete.mar

• Uploaded win32 build

cd /e/builds/moz2_slave/win32_build/build/obj/dist
scp firefox-* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/unsigned/win32/en-US/firefox-3.1b1.zip
cd install/sea
scp firefox-* "cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/unsigned/win32/en-US/Firefox Setup 3.1 Beta 1"
cd ../../update
scp firefox-* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/unsigned/update/win32/en-US/firefox-3.1b1.complete.mar

Repack

• Used slightly modified versions of this script to do l10n: https://bugzilla.mozilla.org/attachment.cgi?id=341938 (had to locally modify because of os-specific problems)
• trimmed ja-JP-mac out on linux & win32, trimmed ja out on mac
• should have 35 locales per platform - verified this with some bash-fu
• This is how I pushed builds - with some extra munging depending on platform:

cd /builds/releases/3.1b1/mozilla-central/obj/dist
mkdir stage
cd stage
mkdir update
cp ../*.bz2 .
cp ../install/*.mar update/
# downloaded rename script from here: http://people.mozilla.org/~bhearsum/misc/rename.pl
perl ../rename.pl .
cd update
perl ../../rename.pl .
# had to do some special munging on mac and win32 because it doesn't name the 'b1' part properly
cd linux-i686
mkdir xpi
cd xpi
cp ../../../install/*.xpi .
for i in *.xpi; do locale=`echo $i | cut -f3 -d.`; mv $i $locale.xpi; done
cd ../../
# now in 'stage' directory again
rsync -av . cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/ # append 'unsigned/' for win32

Sign

instructions

• Had to locally patch sign-release.pl to support betas. bug 408958
• Noticed that checksum-files also generates hashes for unsigned things FIXME bug 459215
  • Manually removed all 'unsigned' entries from the *SUMS files (because they will be trimmed before we release)

L10nVerify

No need to run l10nverify because this is the first release we're shipping locales on.

Generate Updates

• FIXME - hg tools repo should be tagged with UPDATE_PACKAGING tag
• FIXME - ReleaseUpdatesFactory needs to properly handle the '-u' flag to bumping script
• FIXME - make_incremental_updates.py patch in bug 454261 needs to be landed in m-c

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
cvs co -d patcher -r UPDATE_PACKAGING_R6 mozilla/tools/patcher
cvs co -d MozBuild -r UPDATE_PACKAGING_R6 mozilla/tools/release/MozBuild
cvs co -d Bootstrap -r UPDATE_PACKAGING_R6 mozilla/tools/release/Bootstrap/Util.pm
cvs co -d patcher-configs mozilla/tools/patcher-configs
hg clone http://hg.mozilla.org/build/tools
wget -O shipped-locales http://hg.mozilla.org/mozilla-central/raw-file/cf9d1780600f/browser/locales/shipped-locales
perl tools/release/patcher-config-bump.pl -p firefox -v 3.1b1 -a 3.1b1 -o 3.1a2 -b 1 -c patcher-configs/moz191-branch-patcher2.cfg -t stage-old.mozilla.org -f ftp.mozilla.org -d download.mozilla.org -l shipped-locales
# edited patcher config file with a few things:
#  * make <partial> url the same as <complete> url
#  * make 3.1a2 completemarurl the same as 3.1b1 completemarurl (because patcher won't download anything if it can't download the previous builds MARs) - TODO - revert this after snippets are done
#  * delete empty 'past-update' line
cd temp/firefox
ln -s 3.1b1 3.1a2-3.1b1
# manually had to move en-US builds to "proper" location otherwise --create-patchinfo will not work
cd 3.1a2-3.1b1/ftp/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US
cp ../../../../../../../firefox-3.1b1.en-US.linux-i686.complete.mar firefox-3.1b1.complete.mar
cd ../../mac/en-US
cp ../../../../../../../firefox-3.1b1.en-US.mac.complete.mar firefox-3.1b1.complete.mar
cd ../../win32/en-US
cp ../../../../../../../firefox-3.1b1.en-US.win32.complete.mar firefox-3.1b1.complete.mar
cd ../../../../../../../../../../../
perl patcher2.pl --create-patchinfo --partial-patchlist-file=patchlist.cfg  --app=firefox --config=patcher-configs/moz191-branch-patcher2.cfg
# push out snippets
cd temp/firefox/3.1a2-3.1b1
rsync -av -e 'ssh -i ~/.ssh/aus' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20081008-Firefox-3.1b1-test
rsync -av -e 'ssh -i ~/.ssh/aus' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20081008-Firefox-3.1b1
# checked-in the patcher-config used for this release (version 1.3, moz191-branch-patcher2.cfg)
# also checked-in the version to use going forward, which reverts the 'completemarurl' change for 3.1a2

Publish Updates to Test Channels (betatest)

# cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20081008-Firefox-3.1b1 # typo'ed, too late to fix it
~/bin/pushsnip 20081008-Firefox-3.1b1-test