Releases/Thunderbird 2.0.0.18/BuildNotes

Build Engineers

nthomas
Version/config bump/tracking in bug 463284

Bonsai queries

Build 1: cvsroot

Tags

Build 1:

Module	Branch	Tag	Pull date
cvsroot/mozilla	GECKO181_20081030_RELBRANCH	THUNDERBIRD_2_0_0_18_BUILD1	2008-11-05 15:25 PST
l10n/l10n	GECKO181_20081030_RELBRANCH	THUNDERBIRD_2_0_0_18_BUILD1	2008-11-05 15:25 PST

Build data

Type	Build machine
[Windows installer/zip]	production-patrocles
[Mac compressed]	bm-xserve05
[Linux compressed]	production-crazyhorse

Notes

Build 1

Using RELEASE_AUTOMATION_M11, same as for Fx2.0.0.18 and Tb2.0.0.17

Slave cleanup

• production-prometheus-vm
  • need ~10GB for tag, source, update generation and linux verify, stage; already had 20GB free

• production-crazyhorse
  • 23GB free on /builds
  • update tinderbox to automation tag
    • cd /builds/tinderbox/mozilla/tools/tinderbox; cvs up -CPd -r RELEASE_AUTOMATION_M11

• bm-xserve05
  • need ~20GB for mac builds, l10n_verify, mac update verify (but 16GB already allocated for builds); 14G free already
  • update tinderbox to automation tag
    • cd /builds/tinderbox/mozilla/tools/tinderbox; cvs up -CPd -r RELEASE_AUTOMATION_M11

• production-patrocles
  • 53GB free
  • update tinderbox to automation tag
    • cd /cygdrive/e/builds/tinderbox/mozilla/tools/tinderbox; cvs up -CPd -r RELEASE_AUTOMATION_M11

• Land changes and retag tb-moz18-bootstrap.cfg
  • cvs tag -d RELEASE_AUTOMATION_M11 tb-moz18-bootstrap.cfg # to delete the tag on the old version
  • cvs tag RELEASE_AUTOMATION_M11 tb-moz18-bootstrap.cfg # to tag the new version

Setup master before starting:

• update mozilla/tools/buildbot-configs/automation/production/tb-master.cfg for most recent milestone of bootstrap
• as buildmaster@production-1.8-master, cd /builds/buildbot/Thunderbird-Automation/buildbot-configs. Check for local diff, dry run a cvs up, then cvs up
• master.cfg is "Locally Modified" to contain passwords; identical to repository version otherwise.
• Ran 'buildbot start /home/buildmaster/Thunderbird-Automation'
• Buildbot waterfall is here (MPT-VPN); this is a separate instance from the Firefox master, with production-prometheus-vm and bm-xserve05 running a slave for each master
• Stop nightly builds on bm-xserve05 (~cltbld $ buildbot stop macosx-1.8-slave2/) and annotate Mozilla1.8 tree to explain its absence

Start automation run:

buildbot sendchange --username=nthomas --master=localhost:9990 -m"Thunderbird 2.0.0.18 build" release

Tag

Automated, no problems

Source

Automated, no problems

Build/Repack

Automated, no problems

Sign

• Signing doc
• certificate has expired, renewing ...
• signed with renewed cert, no issues -- coop

l10n verification

Automated, no problems found

Generate Updates

Automated, no problems

Publish Updates to Test Channels (betatest & releasetest)

Automated, no problems

Update Verify

Automated, both mac and linux had errors due to AUS not returning updates for the couple of locales. FIXME: Need a 2 minute delay here. Ran both again, but it picked up the failures from the first log again. Moved /builds/release/logs/updates_verify.log to updates_verify.log.1 (mac aliases mv to cp!) - reran and went green.

Push updates to beta channel

$ sudo su - cltbld
$ time ~/bin/backupsnip 20081107-Thunderbird-2.0.0.18-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20081112-1-pre-20081107-Thunderbird-2.0.0.18-beta.tar.bz2 .
real    26m45.116s
user    0m34.449s
sys     1m7.300s

$ time ~/bin/pushsnip 20081107-Thunderbird-2.0.0.18-beta
real    1m34.444s
user    0m0.074s
sys     0m2.976s

Sign Installers

• Done manually using these installer-signing-instructions here
• complete stage-merged:

# on stage
cd /data/cltbld/thunderbird-2.0.0.18/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/thunderbird-2.0.0.18/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:thunderbird .
chmod 644 *SUMS

Update Bouncer

• Manually added entries to download.mozilla.org
• Note for next release: Do not remove the Check Now bit on the Thunderbird-2.0.0.18 Products until well after the change to the rsync module (to prevent the likes of bug 464566)

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage (as cltbld)
rsync -av /data/cltbld/thunderbird-2.0.0.18/stage-merged/ /home/ftp/pub/thunderbird/releases/2.0.0.18/

Update rsync module

On stage.mozilla.org, add the new release to the smaller, faster, rsync module:

# on stage (as cltbld)
cd /pub/mozilla.org/zz/
vi rsyncd-mozilla-current.exclude 

and replace the previous latest release (Tb2.0.0.17) with:

+ thunderbird/releases/2.0.0.18
+ thunderbird/releases/2.0.0.18/**

Final Verification

• Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-thunderbird-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

• Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

• Look for mirrors that do not respond

grep FAIL quickVerify.log

• Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20081107-Thunderbird-2.0.0.18
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20081107-Thunderbird-2.0.0.18-test/$a");'

Publish Updates to Release Channel

• In case you need to cancel/abort updates, the quickest way to do this is here
• While waiting for QA to finish testing, do a backup:

 # on aus2-staging verify we have correct tools
 $ sudo su - cltbld
 $ cd ~/bin
 $ cvs update -dP
 # backup snippets on release/live channel
 $ time ~/bin/backupsnip 20081107-Thunderbird-2.0.0.18
   Running /bin/tar cfvj /opt/aus2/snippets/backup/20081119-1-pre-20081107-Thunderbird-2.0.0.18.tar.bz2 .

   real    29m24.758s
   user    0m27.689s
   sys     0m44.439s

• After QA finished testing the releasetest channel and go from release driver, enable release channel:

 # on aus2-staging
 # put snippets on release/live channel
 $ sudo su - cltbld
 $ time ~/bin/pushsnip 20081107-Thunderbird-2.0.0.18

Update symlink

On stage.m.o, change symlink to point to new release:

 $ cd /pub/mozilla.org/thunderbird/releases
 # confirm where latest symlinks point to where before starting
 $ ls -la latest*
 lrwxrwxrwx  1 cltbld thunderbird 8 Sep 25 15:17 latest -> 2.0.0.17
 lrwxrwxrwx  1 root   thunderbird 8 Feb 26  2008 latest-1.5 -> 1.5.0.14
 lrwxrwxrwx  1 cltbld thunderbird 8 Sep 25 15:17 latest-2.0 -> 2.0.0.17

 $ rm latest    # (note no trailing slash)
 $ ln -s 2.0.0.18 latest
 $ rm latest-2.0   # (note no trailing slash)
 $ ln -s 2.0.0.18 latest-2.0
 # confirm latest symlinks point to newest release
 $ ls -la latest*
 lrwxrwxrwx  1 cltbld thunderbird 8 Nov 19 15:21 latest -> 2.0.0.18
 lrwxrwxrwx  1 root   thunderbird 8 Feb 26  2008 latest-1.5 -> 1.5.0.14
 lrwxrwxrwx  1 cltbld thunderbird 8 Nov 19 15:21 latest-2.0 -> 2.0.0.18