Releases/Thunderbird 2.0.0.16:BuildNotes

Build Engineers

bhearsum
Version/config bump bug

Bonsai queries

Build 1: http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=AviarySuiteBranchTinderbox&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008-07-02+01%3A49+PDT&maxdate=2008-07-02+01%3A51+PDT&cvsroot=%2Fcvsroot

Tags

Build 1:

Module	Branch	Tag	Pull date
cvsroot/mozilla	MOZILLA_1_8_BRANCH	THUNDERBIRD_2_0_0_16_BUILD1 GECKO181_20080612_RELBRANCH
l10n/l10n	MOZILLA_1_8_BRANCH	THUNDERBIRD_2_0_0_16_BUILD1 GECKO181_20080612_RELBRANCH

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]				production-crazyhorse
[Mac compressed]				bm-xserve05
[Linux compressed]				production-patrocles

Notes

Bootstrap Tag: RELEASE_AUTOMATION_M10

See https://bugzilla.mozilla.org/show_bug.cgi?id=440351 for all the setup notes.

Build 1

Tag

No problems.

Source

No problems.

Build

• Build on win32 & linux failed because Bootstrap couldn't be checked out (ssh key error)
  • Changed id_dsa* symlinked to point to cltbld key (they were pointing to tbirdbld). This shouldn't cause problems because other steps (symbols, stage upload) that require keys point explicitly to the tbirdbld key.
• Failed out again because tinderbox-configs directories were checked out with the tbirdbld key.
  • Re-checked them out with the cltbld key and restarted automation.
• Linux build hung when trying to upload Talkback symbols
  • Forgot to copy over production talkback packages. Caught win32 & mac before they hit this. Manually ran the following to upload linux symbols:

cd /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla && tar jxvf /builds/tinderbox/talkback-thunderbird-1.8-linux.tar.bz2 && cp -r fullsoft /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla
cd fullsoft
rm Makefile
cd /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla/fullsoft && /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla/build/autoconf/make-makefile -d ..
make -C /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla/fullsoft fullcircle-push FC_BUILD=2008070711

• Mac Build failed erroneously because 'sendmail' was set wrong in the Bootstrap config (/usr/lib/sendmail instead of /usr/sbin/sendmail)
  • Checked in a bustage fix, continued on.
• Win32 build failed out when trying to upload talkback symbols. Re-ran as follows:

cd /cygdrive/e/builds/tinderbox/Fx-Mozilla1.8-Release
./build-seamonkey.pl --once --mozconfig mozconfig --depend --config-cvsup-dir --skip-mozilla

• • Talkback symbols uploaded fine with this, but builds did not.
    • Didn't know what else to try, so started win32 from scratch.
  • Talkback symbols were fine this time, but I forgot to remove the build logs before re-running. the build failed because it grabbed the wrong buildid
    • Cleared the build logs and respun

Repack

No problems

Sign

Signing doc

L10nVerify

Lots of output from l10nverify. Most of it was from a linux kernel bump on crazyhorse and a EULA change. The build directory also changed from Tb-Mozilla1.8-release to Tb-Mozilla1.8-Release, which caused some more output.

Generate Updates

No problems

Publish Updates to Test Channels (betatest & releasetest)

No problems

Update Verify

• Win32 failed out because win32_logDir was set wrong. (Was /cygdrive/e/builds/release/logs, now /builds/release/logs). Because we run update_verify on production-pacifica-vm, which doesn't have an E: drive, we can't use /cygdrive/e/..
• Re-ran and it worked fine.

Stage

Failed out because 'cltbld' on production-prometheus-vm was not part of the 'thunderbird' group. Re-ran and it worked fine.

Push Updates to Beta Channels

-bash-3.2$ ~/bin/backupsnip 20080709-Thunderbird-2.0.0.16-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20080716-1-pre-20080709-Thunderbird-2.0.0.16-beta.tar.bz2 .
-bash-3.2$ ~/bin/pushsinp 20080709-Thunderbird-3.0.1-beta

Sign Installers

Done manually using these installer-signing-instructions here

• complete stage-merged:

# on stage
cd /data/cltbld/thunderbird-2.0.0.16/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/thunderbird-2.0.0.16/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

• Manually added entries to download.mozilla.org

Update rsync module

On stage.mozilla.org, add the new release to the smaller, faster, rsync module:

# on stage (as cltbld)
cd /pub/mozilla.org/zz/
vi rsyncd-mozilla-current.exclude 

and replace the previous latest release (Tb2.0.0.16) with:

+ thunderbird/releases/2.0.0.16
+ thunderbird/releases/2.0.0.16/**

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage (as cltbld)
rsync -av /data/cltbld/thunderbird-2.0.0.16/stage-merged/ /home/ftp/pub/thunderbird/releases/2.0.0.16/

Final Verification

• Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-thunderbird-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

• Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

• Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20080709-Thunderbird-2.0.0.16
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080709-Thunderbird-2.0.0.16-test/$a");'

Publish Updates to Release Channel

• In case you need to cancel/abort updates, the quickest way to do this is here
• While waiting for QA to finish testing, do a backup:

 # on aus2-staging
 # verify we have correct tools
 $ sudo su - cltbld
 $ cd ~/bin
 $ cvs update -dP
 # backup snippets on release/live channel
 $ sudo su - cltbld
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/backupsnip 20080709-Thunderbird-2.0.0.16

...in progress...

• After QA finished testing the releasetest channel, enabled release channel:

 # on aus2-staging
 # put snippets on release/live channel
 $ sudo su - cltbld
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/pushsnip 20080709-Thunderbird-2.0.0.16