Bootstrap Tag: RELEASE_AUTOMATION_M11
Setup before starting:
- updated master.cfg (no changes found)
- On linux slave (fx-linux-1.9-slave2):
- ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works
- On the mac slave (fx-mac-1.9-slave2):
- removed nothing, as there was loads of space.
- On the windows slave (fx-win32-1.9-slave2):
- Space on slaves before starting:
- fx-linux-1.9-slave2: 20G on /builds
- fx-mac-1.9-slave2: 20G on /
- fx-win32-1.9-slave2: 16.5G on d:, 5.50G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)
- changed frequency of l10n_nightly_scheduler from [1,9,13,17,21] to  in master.cfg
- Kicked off automation:
buildbot sendchange --username=ccooper --master=localhost:9989 -m"Firefox 3.0.6build1 release" go
- Step Tag died right away:
ASSERT: Tag::Execute(): /builds/tags/FIREFOX_3_0_5_BUILD1/cvsroot already exists? at Bootstrap/Step/Tag.pm line 65.
- tagged new bootstrap.cfg as RELEASE_AUTOMATION_M11 and sent another sendchange
- second attempt: no problems
Build & Repack
Publish Updates to Test Channels (betatest & releasetest)
- linux - no problems.
- mac - no problems.
- win32 - similar issues to what we saw with 3.0.5. Ben/Ted rationalized it as follows:
[12:59pm] bhearsum: the chk files are supposed to be forced updates now
[12:59pm] bhearsum: (which means the partial should contain the full file, not just a diff)
[1:00pm] ted: for precisely this reason
[1:00pm] ted: well, sort of
[1:00pm] ted: the files differ between the zip and installer, so the partial was bogus
[1:02pm] bhearsum: the MARs are generated against the zip contents ?
[1:03pm] bhearsum: well, the chk files were certainly forced
[1:03pm] ted: i think so, aren't they?
[1:03pm] ted: isn't that the base reason we hit this problem?
[1:03pm] bhearsum: i'm not entirely certain, to be honest, but it sounds right
[1:03pm] bhearsum: yeah...
[1:03pm] bhearsum: right
[1:04pm] bhearsum: so the update verify test says they're different between the new installer, and the previous installer + mar
[1:04pm] bhearsum: and the reason we force them is so teh MARs don't fail because of this
[1:04pm] bhearsum: ergo, we get these sucky errors in the update verify log, but they won't cause issues
[1:04pm] ted: yeah, so the chk files are still going to differ between the installer+partial and new installer
[1:05pm] bhearsum: even in the complete mar
[1:05pm] ted: but they should be identical in the new zip and old installer+partial
[1:05pm] bhearsum: right
Push updates to beta channel
# put snippets on beta
$ sudo su - cltbld
# make sure using latest version of scripts in mozilla/tools/release/bin/
$ cd bin
$ cvs update .
$ cd /opt/aus2/snippets/staging
# note the required parameter must match what will be used with pushsnip below.
$ time ~/bin/backupsnip 20090120-Firefox-3.0.6-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20090127-1-pre-20090120-Firefox-3.0.6-beta.tar.bz2 .
$ time ~/bin/pushsnip 20090120-Firefox-3.0.6-beta
Running /usr/bin/rsync -PaO /opt/aus2/snippets/staging/20090120-Firefox-3.0.6-beta/ /opt/aus2/incoming/3
Done manually using these installer-signing-instructions here.
# on stage
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
- Create MD5 and SHA1 checksum files
# on stage
- Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS
- Note for next release: Do not remove the Check Now bit on the Firefox-3.0.6 Products until well after the change to the rsync module (to prevent the likes of bug 464566)
Push to mirrors
- push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0.6/stage-merged/ /home/ftp/pub/firefox/releases/3.0.6/
- edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.6) and remove the previous release (3.0.5).
- Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
Publish Updates to Release Channel
While waiting for the formal "go," ran:
-bash-3.2$ time ~/bin/backupsnip 20090120-Firefox-3.0.6
Running /bin/tar cfvj /opt/aus2/snippets/backup/20090203-1-pre-20090120-Firefox-3.0.6.tar.bz2 .
After formal go, ran:
$ time ~/bin/pushsnip 20090120-Firefox-3.0.6
- fix latest symlink
rm latest-3.0 && ln -s 3.0.6 latest-3.0
After release, there were issues with several mirrors. nthomas had to disable all updates while he and IT investigated.
For those curious, the issue we had that caused us to pull updates was that a db server was mistakenly(?) taken out of rotation for bouncer, which was causing partial updates to fail. IT is investigating why this happened. Leaving updates on the release channel for users would've meant all (or maybe just most) users receiving full updates -- a problem for users (large download) and a problem for mirrors (lots more bandwidth). We'll talk more about this at the post-mortem that I'll schedule tomorrow for either later this week or early next.