MozillaWiki

Thunderbird 2.0.0.14:BuildNotes

Revision as of 20:32, 22 April 2008 by Rtessner (talk | contribs) (→‎l10n-Verify)

Build Engineers

nthomas, rick

Version/config bump bug

bug 426259 Morphed from the cancelled Tb2.0.0.13 build

Bonsai queries

Checking for no changes on the rel branch since Fx20014

Tags

Using the GECKO181_20080311_RELBRANCH.

Module Tag Pull date
cvsroot/mozilla THUNDERBIRD_2_0_0_14_RELEASE GECKO181_20080311_RELBRANCH @ 2008-04-17 13:27 PDT
THUNDERBIRD_2_0_0_14_RC1 GECKO181_20080311_RELBRANCH @ 2008-04-17 13:27 PDT
l10n/l10n THUNDERBIRD_2_0_0_14_RELEASE GECKO181_20080311_RELBRANCH @ 2008-04-17 13:42 PDT
THUNDERBIRD_2_0_0_14_RC1 GECKO181_20080311_RELBRANCH @ 2008-04-17 13:42 PDT

NB: Talkback not listed, we're using pre-built binaries (bug 403701). It was still tagged because this change is very new, and the automation needs remedial lessons.

Build data

See page history for BuildIDs of prior RCs

Type Build ID SHA1 Push date Build machine
[Windows installer] RC1: patrocles (cerberus-vm for l10n)
[Mac compressed] RC1: xserve02
[Linux compressed] RC1: crazyhorse (karma for l10n)

Automation

Bootstrap tag: RELEASE_AUTOMATION_M8_1

Setup before starting:

  • used existing version bump bug to attached diff for tb-moz18-bootstrap.cfg
  • revert mozilla/config/milestone.txt and mozilla/client.mk on the rel branch so that Tagging can substitute cleanly (FIXME: need bug)
  • ensure that machines have enough resources (bug 393274)
    • On production-1.8-master, remove private and public areas:
      • rm -rf /data/cltbld/firefox-* (check with folks responsible for the releases being rm'd before removing ... in this case it was nthomas and bhearsum)
      • rm -rf /builds/tags/* - did not do this, plenty of space (and handy to have copies of what was tagged to work around CVS's lack of atomicity of tagging)
      • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
      • pull in 2.0.0.12 (previous release) files with rsync -av stage.mozilla.org:/home/ftp/pub/thunderbird/nightly/2.0.0.12-candidates /data/ftp/pub/thunderbird/nightly/
    • On the build machines (see table above), checked that there was sufficient working space (~ 1GB), and stopped tinderbox. Also verified there is space available on bm-xserve04 for l10n-verify.
    • On prometheus-vm, checked that there was space for another update run (4GB)

The following bits, to the end of this Automation section, still need to be done. Delayed due to ssh key issues on the various tinderbox clients. Will do the Tag and Source steps tho. 2008-04-17.

  • on each machine, setup release automation using the tag above
cd /builds/release   # or /cygdrive/X/builds/release
cvs -nq up -r RELEASE_AUTOMATION_M8_1
# check for merges (M) or conflicts (C); if any found, check for local changes
cvs -q up -r RELEASE_AUTOMATION_M8_1
cd configs
cvs up -A     # in case we need to make changes later
cd ..
rm bootstrap.cfg logs
ln -s configs/tb-moz18-bootstrap.cfg bootstrap.cfg
mkdir -p old-logs/logs.tb20014/rc1
ln -s old-logs/logs.tb20014/rc1 logs

make test

Some machines don't have the perl module Config::General so comment out 

use Config::General;

at the top of Bootstrap/Step/PatcherConfig.pm. Only need this on prometheus-vm anyway, which has it.

Tag

On production-1.8-master in /builds/release: 

./release -o Tag 2>&1 | tee logs/release-Tag-RC1.log

Source

On production-1.8-master in /builds/release: 

./release -o Source 2>&1 | tee logs/release-Source-RC1.log

Failed with message 

build-console.build.mozilla.org - Step Source died: No such config variable:
externalStagingUser at Bootstrap/Config.pm line 93.

Appears to be related to this configuration change which removed the externalStagingUser variable.

However, tools/release/Bootstrap/Util.pm on the RELEASE_AUTOMATION_M8_1 tag still references that variable. There were changes made on the trunk for Util.pm that removed those references.

Resolved with this commit which restored the external* variables.

This change will need to be undone once the thunderbird 2.0.0.14 release is complete.

The Source step was completed by doing the following on production-1.8-master 

 cd /builds/release/configs
 cvs up
 cd ..
  ./release -o Source -p 2>&1 | tee logs/release-Source-RC1-Push.log

This completed successfully.

Build

There ssh key issues to workaround - the nightly builders were converted to tbirdbld but releases still use cltbld. On each box,

  • shut down tinderbox builds and add a comment on the Mozilla1.8 tree
  • in ~/.ssh
rm id_dsa; ln -s cltbld_dsa id_dsa
rm id_dsa.pub; ln -s cltbld_dsa.pub id_dsa.pub
  • update /builds/tinderbox/Tb-Mozilla1.8-Release/tinderbox-configs/CVS/Root so that it uses the cltbld account (just for this release, we'll leave it on cltbld)
  • update Bootstrap as in the Automation section

On Windows, check that lines 43-49 of Build.pm are commented out, since Cygwin complains "Permissied denied" trying to execute those lines. The disks are already mounted in binmode.


Then start the build, by changing directory to /builds/release (prefix with /cygdrive/e/ on Windows) and 

./release -o TinderConfig 2>&1 | tee logs/release-TinderConfig.log
./release -o Build 2>&1 | tee logs/release-Build.log


After the build is done, and if the tinderbox is not needed for l10n Repacks (eg patrocles, crazyhorse),

  • reverse the changes in ~/.ssh. NB: bm-xserve02 uses ffxbld, others use tbirdbld. Eg
rm id_dsa; ln -s tbirdbld_dsa id_dsa
rm id_dsa.pub; ln -s tbirdbld_dsa.pub id_dsa.pub
  • restart tinderbox
cd /builds/tinderbox
./tinderbox multi start

Repack

On the tinderboxes in the table above

  • stop tinderbox
  • do the ssh key changes as in the Build section
  • update Bootstrap as in the Automation section
./release -o Repack -e 2>&1 | tee logs/release-Repack-Execute.log
./release -o Repack -p 2>&1 | tee logs/release-Repack-Push.log
./release -o Repack -a 2>&1 | tee logs/release-Repack-Announce.log
karma bm-xserve02 cerberus-vm
Repack -e ok ok ok
Repack -p ok not ok 
file has vanished: "/home/ftp/pub/thunderbird/nightly/2.0.0.14-candidates/rc1/.thunderbird-2.0.0.14.pa-IN.linux-i686.complete.mar.KwvZoP"
not ok 
rsync: link_stat "/data/cltbld/thunderbird-2.0.0.12/." failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(702)
Repack -a ok

Analysis/Fixes (nthomas):

  • Linux
    • Execute uploaded to cltbld@production-1.8-master.build.mozilla.org:/home/ftp/pub/thunderbird/nightly/2008-04-21-13-thunderbird2.0.0.14-l10n/. ie not .../2008/04/2008-04-21-13-thunderbird2.0.0.14-l10n/. This is because the tinderbox code was not symlinked properly in karma:/builds/tinderbox/Tb-Mozilla1.8-l10n-Release - effectively it was using older code. This would have been a long standing issue.
    • Fixed the builds on the master by moving the linux files into .../2008/04/2008-04-21-13-thunderbird2.0.0.14-l10n/ and replacing .../thunderbird/nightly/2008-04-21-13-thunderbird2.0.0.14-l10n/ with a symlink into 2008/04/.
    • Fixed tinderbox setup (backup in bak-20080422)
    • To verify all files are present, ran os=linux; ~/check_locales.sh /home/ftp/pub/thunderbird/nightly/2.0.0.14-candidates/rc1/ ${os} ~/thunderbird-2.0.0.14/batch-source/rc1/mozilla/mail/locales/shipped-locales on stage.m.o. Got no complaints about missing files.
    • Fixed up ~/.ssh symlinks (using ffxbld key), and restarted tinderbox
  • Mac
    • Execute uploaded to the correct location (.../2008/04/2008-04-21-13-thunderbird2.0.0.14-l10n/)
    • Push failed to move any mac files into the candidates direcotry because it was using /home/ftp/pub/thunderbird/nightly/2008-04-21-13-thunderbird2.0.0.14-l10n/ as the source. So it got the Linux dir instead of a symlink to the mac dir.
    • The error file has vanished:"/home/ftp/pub/thunderbird/nightly/2.0.0.14-candidates/rc1/.thunderbird-2.0.0.14.pa-IN.linux-i686.complete.mar.KwvZoP" comes from the Linux Push running simultaneously with the Mac one (ie Linux was pushing into the candidates dir as Mac was syncing with stage, that's a rsync temp file). Ignoring this as harmless.
    • Ran Push again, which copied the Mac files properly, but got rsync error: some files could not be transferred (code 23) at main.c(702) when doing SyncToStage. Should have applied the .../thunderbird/nightly/2008-04-21-13-thunderbird2.0.0.14-l10n fix on stage too.
    • After fixing stage, ran rsync -av /home/ftp/pub/thunderbird/nightly/ cltbld@stage-old.mozilla.org:/home/ftp/pub/thunderbird/nightly/ on the master, and only the symlink was copied over.
    • the equivlanet check_locales.sh now says all mac files are present on stage
    • Fixed up ~/.ssh symlinks (using ffxbld key), and restarted tinderbox
  • Windows
    • Execute uploaded to the correct location
    • Push copied the bits to /home/ftp/pub/thunderbird/nightly//2.0.0.12-candidates/rc1/unsigned/. Turns out tb-moz18-bootstrap.cfg was on the RELEASE_AUTOMATION_M8_1 tag rather than the tip.
    • On the master, cleaned out /home/ftp/pub/thunderbird/nightly/2.0.0.12-candidates/rc1/unsigned of 2.0.0.14 files, including all of windows-xpi/.
    • Did the same on stage, and restored the correct xpi's from 2008/02/2008-02-13-09-thunderbird2.0.0.12-l10n/windows-xpi
    • Updated the bootstrap configs/ dir to tip, moved logs/* to logs/attempt1/, and reran Repack Execute, Push, and Announce
    • Fixed up ~/.ssh symlinks (using ffxbld key), and restarted tinderbox

Sign

l10n-Verify

Ran this on bm-xserve05 (via vnc) 

./release -o Repack -v 2>&1 | tee logs/release-Repack-Verify.log

This was run a first time simply by ssh'ing into bm-xserve05. A screen session was fired up and the above Repack command was run. That failed with various hdiutil errors. It turns out that hdiutil will not work properly under a screen session.

Retrieved from "https://wiki.allizom.org/index.php?title=Thunderbird_2.0.0.14:BuildNotes&oldid=91157"