Releases/Firefox 3.1b1:BuildNotes

Build Engineers

bhearsum
release tracking bug

Signed-off Revision

http://hg.mozilla.org/mozilla-central/rev/197f83ad7678

Tags

Build 1:

Module	Tag	Changeset
mozilla-central		http://hg.mozilla.org/mozilla-central/rev/197f83ad7678

All l10n repositories listed in l10n-changesets were tagged with FIREFOX_3_1b1_RELEASE: http://hg.mozilla.org/build/buildbot-configs/file/FIREFOX_3_1b1_RELEASE/mozilla2/l10n-changesets

Build data

Type	Build ID	Build machine
[Windows installer/zip]	20081007144708	moz2-win32-slave11
[Mac compressed]	20081007125523	bm-xserve19
[Linux compressed]	20081007112159	moz2-linux-slave05

Notes

• Cleared up disk space before starting.
  • Removed old actionmonkey directories on linux slaves
• Free space before starting:
  • moz2-linux-slave01: 13G free
  • moz2-linux-slave02: 16G free
  • moz2-linux-slave03: 16G free
  • moz2-linux-slave05: 12G free
  • moz2-linux-slave06: 16G free
  • bm-xserve16: 25Gi free
  • bm-xserve17: 28Gi free
  • bm-xserve18: 25Gi free
  • bm-xserve19: 23Gi free
  • moz2-win32-slave01: 16G free
  • moz2-win32-slave02: 17G free
  • moz2-win32-slave03: 17G free
  • moz2-win32-slave05: 16.5G free
  • moz2-win32-slave06: 15.5G free
  • moz2-win32-slave11: 22G free
  • moz2-win32-slave12: 18G free
  • moz2-win32-slave13: 22G free

Build 1

Tag

• Tag failed out with a 'setupBuild' exception. Buildbot cannot handle Builders with more than 100 ShellCommands, see this line: http://mxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/process/base.py#397
  • Fixed by changing that line to '< 1000'
  • Upstream buildbot bug: http://buildbot.net/trac/ticket/366
  • Our bug: bug 458911
• Tag failed again because the 'hg.mozilla.org' host key wasn't accepted on the production slaves.
  • Went around and accepted the key on linux slaves 01, 02, 03, 05, 06, 11, 12 (mac and win32 machines do not push to hg)
• Tag failed again with a traceback when trying to push:

pushing to ssh://hg.mozilla.org/l10n-central/be
searching for changes
** unknown exception encountered, details follow
** report bug details to http://www.selenic.com/mercurial/bts
** or mercurial@selenic.com
** Mercurial Distributed SCM (version 0.9.5)
Traceback (most recent call last):
 File "/tools/python/bin/hg", line 14, in <module>
   mercurial.dispatch.run()
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 20, in run
   sys.exit(dispatch(sys.argv[1:]))
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 29, in dispatch
   return _runcatch(u, args)
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 45, in _runcatch
   return _dispatch(ui, args)
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 348, in _dispatch
   ret = _runcommand(ui, options, cmd, d)
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 401, in _runcommand
   return checkargs()
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 357, in checkargs
   return cmdfunc()
 File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 340, in <lambda>
   d = lambda: func(ui, repo, *args, **cmdoptions)
 File "/tools/python/lib/python2.5/site-packages/mercurial/commands.py", line 2147, in push
   r = repo.push(other, opts['force'], revs=revs)
 File "/tools/python/lib/python2.5/site-packages/mercurial/localrepo.py", line 1385, in push
   return self.push_unbundle(remote, force, revs)
 File "/tools/python/lib/python2.5/site-packages/mercurial/localrepo.py", line 1465, in push_unbundle
   return remote.unbundle(cg, remote_heads, 'push')
 File "/tools/python/lib/python2.5/site-packages/mercurial/sshrepo.py", line 195, in unbundle
   l = int(self.pipei.readline())
ValueError: invalid literal for int() with base 10: 
remote: abort: could not lock repository /repo/hg/mozilla/l10n-central/be:  Permission denied
program finished with exit code 1

• • Turns out that we never asked for 'ffxbld' to have push access to l10n repositories. Filed bug 458908 to get that enabled.

• Tagging went fine after this

• Two action items:
  • Dependent scheduler did not fire TODO bug 459213
  • Check-in comments say 'cltbld' when they should say 'ffxbld' TODO bug 459214

Source

• Automated, no problems.
• Manually pushed builds to stage:

# as cltbld@stage.mozilla.org
cd /home/ftp/pub/firefox/nightly
mkdir -p 3.1b1-candidates/build1
cd 3.1b1-candidates/build1
mkdir linux-i686 mac win32 update source
# as cltbld@moz2-linux-slave06 (where the source tarball/bundle were generated
cd /builds/moz2_slave/source
scp firefox-3.1b1* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/source

Build

• No problems with linux build
• Created *_info.txt files on stage
• Created en-US subdirs on stage:

cd /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1
mkdir linux-i686/en-US mac/en-US win32/en-US

• On moz2-linux-slave05:

cd /builds/moz2_slave/linux_build/build/obj-firefox
make package
make -C tools/update-packaging
# upload
scp firefox-3.1b1.en-US.linux-i686.tar.bz2 cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/linux-i686/en-US/firefox-3.1b1.tar.bz2
cd update
scp firefox-* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US/firefox-3.1b1.complete.mar

• Win32 and mac builds got interrupted when the Buildbot master hung because of graph server downtime. bug 458939
  • when the master came back up they were restarted
• A mac nightly build got uploaded by mistake (bug 458968), redid on bm-xserve19 with

# the old one moved to stage:~cltbld/wrong-mac-en-US-firefox-3.1b1.dmg
cd /builds/moz2_slave/macosx_build/build/obj-firefox/ppc
make package
make -C tools/update-packaging
scp dist/firefox-3.1b1.en-US.mac.dmg cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/mac/en-US/firefox-3.1b1.dmg
cd update
scp firefox-* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/update/mac/en-US/firefox-3.1b1.complete.mar

• Uploaded win32 build

cd /e/builds/moz2_slave/win32_build/build/obj/dist
scp firefox-* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/unsigned/win32/en-US/firefox-3.1b1.zip
cd install/sea
scp firefox-* "cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/unsigned/win32/en-US/Firefox Setup 3.1 Beta 1"
cd ../../update
scp firefox-* cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/unsigned/update/win32/en-US/firefox-3.1b1.complete.mar

Repack

• Used slightly modified versions of this script to do l10n: https://bugzilla.mozilla.org/attachment.cgi?id=341938 (had to locally modify because of os-specific problems)
• trimmed ja-JP-mac out on linux & win32, trimmed ja out on mac
• should have 35 locales per platform - verified this with some bash-fu
• This is how I pushed builds - with some extra munging depending on platform:

cd /builds/releases/3.1b1/mozilla-central/obj/dist
mkdir stage
cd stage
mkdir update
cp ../*.bz2 .
cp ../install/*.mar update/
# downloaded rename script from here: http://people.mozilla.org/~bhearsum/misc/rename.pl
perl ../rename.pl .
cd update
perl ../../rename.pl .
# had to do some special munging on mac and win32 because it doesn't name the 'b1' part properly
cd linux-i686
mkdir xpi
cd xpi
cp ../../../install/*.xpi .
for i in *.xpi; do locale=`echo $i | cut -f3 -d.`; mv $i $locale.xpi; done
cd ../../
# now in 'stage' directory again
rsync -av . cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/ # append 'unsigned/' for win32

Sign

instructions

• Had to locally patch sign-release.pl to support betas. bug 408958
• Noticed that checksum-files also generates hashes for unsigned things FIXME bug 459215
  • Manually removed all 'unsigned' entries from the *SUMS files (because they will be trimmed before we release)

L10nVerify

No need to run l10nverify because this is the first release we're shipping locales on.

Generate Updates

• FIXME - hg tools repo should be tagged with UPDATE_PACKAGING tag
  • Tagged hg.m.o/build/tools with UPDATE_PACKAGING_R6 for future use
• FIXME - ReleaseUpdatesFactory needs to properly handle the '-u' flag to bumping script bug 459216
• FIXME - make_incremental_updates.py patch in bug 454261 needs to be landed in m-c

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
cvs co -d patcher -r UPDATE_PACKAGING_R6 mozilla/tools/patcher
cvs co -d MozBuild -r UPDATE_PACKAGING_R6 mozilla/tools/release/MozBuild
cvs co -d Bootstrap -r UPDATE_PACKAGING_R6 mozilla/tools/release/Bootstrap/Util.pm
cvs co -d patcher-configs mozilla/tools/patcher-configs
hg clone http://hg.mozilla.org/build/tools
wget -O shipped-locales http://hg.mozilla.org/mozilla-central/raw-file/cf9d1780600f/browser/locales/shipped-locales
perl tools/release/patcher-config-bump.pl -p firefox -v 3.1b1 -a 3.1b1 -o 3.1a2 -b 1 -c patcher-configs/moz191-branch-patcher2.cfg -t stage-old.mozilla.org -f ftp.mozilla.org -d download.mozilla.org -l shipped-locales
# edited patcher config file with a few things:
#  * make <partial> url the same as <complete> url
#  * make 3.1a2 completemarurl the same as 3.1b1 completemarurl (because patcher won't download anything if it can't download the previous builds MARs) - TODO - revert this after snippets are done
#  * delete empty 'past-update' line
cd temp/firefox
ln -s 3.1b1 3.1a2-3.1b1
# manually had to move en-US builds to "proper" location otherwise --create-patchinfo will not work
cd 3.1a2-3.1b1/ftp/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US
cp ../../../../../../../firefox-3.1b1.en-US.linux-i686.complete.mar firefox-3.1b1.complete.mar
cd ../../mac/en-US
cp ../../../../../../../firefox-3.1b1.en-US.mac.complete.mar firefox-3.1b1.complete.mar
cd ../../win32/en-US
cp ../../../../../../../firefox-3.1b1.en-US.win32.complete.mar firefox-3.1b1.complete.mar
cd ../../../../../../../../../../../
perl patcher2.pl --create-patchinfo --partial-patchlist-file=patchlist.cfg  --app=firefox --config=patcher-configs/moz191-branch-patcher2.cfg
# push out snippets
cd temp/firefox/3.1a2-3.1b1
rsync -av -e 'ssh -i ~/.ssh/aus' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20081008-Firefox-3.1b1-test
rsync -av -e 'ssh -i ~/.ssh/aus' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20081008-Firefox-3.1b1
# checked-in the patcher-config used for this release (version 1.3, moz191-branch-patcher2.cfg)
# also checked-in the version to use going forward, which reverts the 'completemarurl' change for 3.1a2

Publish Updates to Test Channels (betatest)

# cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20081008-Firefox-3.1b1 # typo'ed, too late to fix it
~/bin/pushsnip 20081008-Firefox-3.1b1-test

Update Verify

Note that update verify only tests old versions, so only en-US was tested:

bm-xserve01:/builds/releases/3.1b1/update_verify/updates cltbld$ ./verify.sh -c moz191-firefox-mac.cfg       
Using config file moz191-firefox-mac.cfg
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a2/20080829065003/Darwin_Universal-gcc3/en-US/betatest/update.xml
07:29:15 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/mac/en-US/firefox-3.1b1.complete.mar [16903884/16903884] -> "update/partial.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a2/20080829065003/Darwin_Universal-gcc3/en-US/betatest/update.xml
07:29:16 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/mac/en-US/firefox-3.1b1.complete.mar [16903884/16903884] -> "update/complete.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a1/2008072510/Darwin_Universal-gcc3/en-US/betatest/update.xml
07:29:17 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/mac/en-US/firefox-3.1b1.complete.mar [16903884/16903884] -> "update/partial.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a1/2008072510/Darwin_Universal-gcc3/en-US/betatest/update.xml
07:29:18 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/mac/en-US/firefox-3.1b1.complete.mar [16903884/16903884] -> "update/complete.mar" [1]
bm-xserve01:/builds/releases/3.1b1/update_verify/updates cltbld$ ./verify.sh -c moz191-firefox-win32.cfg
Using config file moz191-firefox-win32.cfg
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a2/20080829082037/WINNT_x86-msvc/en-US/betatest/update.xml
07:29:30 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/win32/en-US/firefox-3.1b1.complete.mar [10307200/10307200] -> "update/partial.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a2/20080829082037/WINNT_x86-msvc/en-US/betatest/update.xml
07:29:31 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/win32/en-US/firefox-3.1b1.complete.mar [10307200/10307200] -> "update/complete.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a1/2008072308/WINNT_x86-msvc/en-US/betatest/update.xml
07:29:34 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/win32/en-US/firefox-3.1b1.complete.mar [10307200/10307200] -> "update/partial.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a1/2008072308/WINNT_x86-msvc/en-US/betatest/update.xml
07:29:35 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/win32/en-US/firefox-3.1b1.complete.mar [10307200/10307200] -> "update/complete.mar" [1]
bm-xserve01:/builds/releases/3.1b1/update_verify/updates cltbld$ ./verify.sh -c moz191-firefox-linux.cfg
Using config file moz191-firefox-linux.cfg
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a2/20080829071937/Linux_x86-gcc3/en-US/betatest/update.xml
07:29:40 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US/firefox-3.1b1.complete.mar [9610314/9610314] -> "update/partial.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a2/20080829071937/Linux_x86-gcc3/en-US/betatest/update.xml
07:29:41 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US/firefox-3.1b1.complete.mar [9610314/9610314] -> "update/complete.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a1/2008072306/Linux_x86-gcc3/en-US/betatest/update.xml
07:29:43 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US/firefox-3.1b1.complete.mar [9610314/9610314] -> "update/partial.mar" [1]
Using  https://aus2.mozilla.org/update/1/Firefox/3.1a1/2008072306/Linux_x86-gcc3/en-US/betatest/update.xml
07:29:44 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US/firefox-3.1b1.complete.mar [9610314/9610314] -> "update/complete.mar" [1]

Update Bouncer

Manually added entries to download.mozilla.org

zh-CN build2

• Localizer landed fix on the release branch: http://hg.mozilla.org/l10n-central/zh-CN
• Tagged that changeset as FIREFOX_3_1b1_BUILD2 and FIREFOX_3_1b1_RELEASE
• re-generated zh-CN builds on all platforms (no logs, sorry).
• Pushed them to cltbld@stage.mozilla.org:/home/.../3.1b1-candidates/build2
• Symlinked everything but signed win32 builds in (those will be done post zh-CN build2 signing):

1087  for i in `ls ../../build1/linux-i686`; do ln -s ../../build1/linux-i686/$i; done
for i in `ls ../../../build1/linux-i686`; do ln -s ../../../build1/linux-i686/$i; done
for i in `ls ../../build1/mac`; do ln -s ../../build1/mac/$i; done
for i in `ls ../../../build1/mac`; do ln -s ../../../build1/mac/$i; done
for i in `ls ../../../../build1/unsigned/update/win32`; do ln -s ../../../../build1/unsigned/update/win32/$i; done
for i in `ls ../../../build1/unsigned/win32`; do ln -s ../../../build1/unsigned/win32/$i; done
for i in `ls ../../../../build1/unsigned/win32/xpi`; do ln -s ../../../../build1/unsigned/win32/xpi/$i; done
history | grep for

sign zh-CN build2

• Followed instructions with these tweaks:
• Only pulled win32 en-US (build1) and linux/mac/win32 zh-CN (build2) into unsigned-build2:

rsync -av -e ssh  --include=linux-i686/ --include=mac/ --include=unsigned/ --include=*irefox* --include=win32  --include=*zh-CN* --include=update/ --exclude=* cltbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ unsigned-build${BUILD}
rsync -av -e ssh --include=unsigned/ --include=*irefox* --include=win32  --include=*en-US* --include=update/ --exclude=* cltbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build1/ unsigned-build${BUILD}

• Ran sign-release.pl in previously signed mode (-p)
• After running sign-release.pl trimmed out en-US builds from signed-build2 before generating detached sigs and pushing back (because there's no need to do either of these for en-US on build2)
• Pushed builds back
• Regenrated MD5SUMS and SHA1SUMS (as per instructions)
  • Had to add '-follow' to ~/bin/checksum-files:sub gather_files() to make sure non-zh-CN files got sums generated - reverted that change afterwards

Push to mirrors

This is a bit more complicated than usual because we did a respin for zh-CN. Did the following before pushing live:

# cltbld@stage
cd /data/cltbld
# copy everything from build1
rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/ firefox-3.1b1/
# copy only zh-CN from build2
rsync -av --include=win32 --include=mac --include=linux-i686 --include=[Ff]irefox* --include=*zh-CN* --exclude=* /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/ firefox-3.1b1/
cd firefox-3.1b1
-bash-3.00$ md5sum mac/ga-IE/Firefox\ 3.1\ Beta\ 1.dmg
610cb5adb38d76fa251eda8a777a5d49  mac/ga-IE/Firefox 3.1 Beta 1.dmg
-bash-3.00$ md5sum /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/mac/ga-IE/Firefox\ 3.1\ Beta\ 1.dmg
610cb5adb38d76fa251eda8a777a5d49  /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/mac/ga-IE/Firefox 3.1 Beta 1.dmg

-bash-3.00$ for i in linux-i686 mac win32; do md5sum $i/zh-CN/[Ff]irefox* | grep -v asc; md5sum /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/$i/zh-CN/[Ff]irefox* | grep -v asc; done
c7f357cc3c43e4310fba7381f565a593  linux-i686/zh-CN/firefox-3.1b1.tar.bz2
c7f357cc3c43e4310fba7381f565a593  /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/linux-i686/zh-CN/firefox-3.1b1.tar.bz2
5d41dea708e0a9d85990e85f84a4958a  mac/zh-CN/Firefox 3.1 Beta 1.dmg
5d41dea708e0a9d85990e85f84a4958a  /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/mac/zh-CN/Firefox 3.1 Beta 1.dmg
58b6c8f5bcf83c4cb80e5b7d727f8930  win32/zh-CN/Firefox Setup 3.1 Beta 1.exe
58b6c8f5bcf83c4cb80e5b7d727f8930  /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/win32/zh-CN/Firefox Setup 3.1 Beta 1.exe