Releases/Thunderbird 10.0.5esr/BuildNotes: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(31 intermediate revisions by 2 users not shown)
Line 4: Line 4:
= Bugs hit =
= Bugs hit =
Enter any bugs pre-existing or newly discovered and filed during the release:
Enter any bugs pre-existing or newly discovered and filed during the release:
* {{bug|}} - description
* {{bug|760353}} - make source-package fails on comm-esr10 because MKDIR is undefined
*
* {{bug|760361}} - l10n repacks failing in configure
* <strike>{{bug|760385}} - we clobber hg shares more frequently than necessary</strike> dupe of {{bug|748158}}
* {{bug|760389}} - some builders not clobbered on build2
* {{bug|760390}} - en-US builds not pulling right mozilla-esr10 revision
* {{bug|760397}} - source builder not getting right mozilla-esr10 revision
* {{bug|760807}} - Program Compatibility Assistant shown on Windows 7 when setting as default mail client (NSIS version)
* {{bug|761951}} - not checking releasetest in final verification
* {{bug|763769}} - release sanity complains about mozconfig differences


=Build Engineers=
=Build Engineers=
Line 11: Line 18:


=Signed-off Revision(s)=
=Signed-off Revision(s)=
* comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/570c22432c13
 
* mozilla-esr10: http://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd
L10N changesets: unchanged
* L10N changesets: unchanged
 
Build1:
* comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/570c22432c13 on default
* mozilla-esr10: http://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH
 
Build2:
* comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/48677c4f453a on GECKO1005_2012053119_RELBRANCH
* mozilla-esr10: http://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH
 
Build 3:
* comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/939269a86b88 on GECKO1005_2012053119_RELBRANCH
* mozilla-esr10: https://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH


=Build data=
=Build data=
Line 26: Line 44:
| rowspan="4" | 1  
| rowspan="4" | 1  
| Linux
| Linux
| rowspan="4" |  
| rowspan="4" | 20120531193048
|  
| mv-moz2-linux-ix-slave09
|  
| 38 mins, 47 secs
|-
|-
| Linux64
| Linux64
|  
| bld-centos5-64-vmw-003
|  
| 45 mins, 40 secs
|-
|-
| Mac
| Mac
|  
| moz2-darwin10-slave44
|  
| 3 hrs, 32 mins, 31 secs
|-
|-
| Windows  
| Windows  
|  
| mw32-ix-slave26
|  
| 1 hrs, 43 mins, 28 secs
|-
| rowspan="1" | 2
| Linux
| rowspan="1" | 20120601013708
| mv-moz2-linux-ix-slave09
| 38 mins, 13 secs
|-
| rowspan="4" | 3
| Linux
| rowspan="4" | 20120601041309
| mv-moz2-linux-ix-slave09
| 37 mins, 26 secs
|-
| Linux64
| bld-centos5-64-vmw-003
| 41 mins, 53 secs
|-
| Mac
| moz2-darwin10-slave44
| 3 hrs, 32 mins, 5 secs
|-
| Windows
| mw32-ix-slave02
| 1 hrs, 39 mins, 32 secs
|-
| 3 Spin 2
| Windows
| 20120605154056
| mw32-ix-slave02
| 1 hrs, 39 mins, 18 secs
|}
|}


Line 85: Line 133:
=== E-mail mirrors@mozilla.org ===
=== E-mail mirrors@mozilla.org ===
N/A
N/A
<font color='red'>'''DONE TO HERE'''</font>
<br/><font color='red'>'''NOTES TRIMMED TO HERE'''</font>


=== Edit rsync exclude files ===
=== Edit rsync exclude files ===
Edit /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude on stage so that it excludes the current release. This prevents external mirrors from picking up the new release prematurely.
N/A
 
For rapid release betas, this is not necessary as there's already an exclude that matches beta releases.


=== Tag ===
=== Tag ===
Line 101: Line 144:


=== Source ===
=== Source ===
 
Failed:
<pre>
make source-package hg-bundle HG_BUNDLE_REVISION=7c765c397d25
in dir /builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb (timeout 2700 secs)
make[1]: Entering directory `/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb/mail/installer'
Packaging source tarball...
p ../../mozilla/dist/source/
make[1]: p: Command not found
make[1]: [source-package] Error 127 (ignored)
(cd /builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10 && gtar -c --owner=0 --group=0 --numeric-owner --mode="go-w" --exclude=".hg*" --exclude="CVS" --exclude=".cvs*" --exclude=".mozconfig*" --exclude="*.pyc" --exclude="/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/mozilla/Makefile" --exclude="/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/mozilla/dist" --exclude="objdir-tb" -f - ../comm-esr10) | bzip2 -vf > ../../mozilla/dist/source/thunderbird-10.0.5esr.source.tar.bz2
/bin/sh: ../../mozilla/dist/source/thunderbird-10.0.5esr.source.tar.bz2: No such file or directory
gtar: Removing leading `../' from member names
make[1]: *** [source-package] Error 1
make[1]: Leaving directory `/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb/mail/installer'
make: *** [source-package] Error 2
</pre>
{{Bug|760353}} filed.


=== Build ===
=== Build ===
==== Firefox ====
==== en-US ====
No problems.


===== Firefox repacks =====
==== Repacks ====
Failing in configure, {{bug|760361}} filed on:
<pre>/builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/configure
Adding configure options from /builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/.mozconfig:
  --with-l10n-base=/builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/obj-l10n/../../mozilla-esr10
  --enable-application=mail
  --enable-official-branding
  --enable-update-channel=esr
  --enable-update-packaging
creating cache ./config.cache
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking build system type... x86_64-unknown-linux-gnu
checking for gawk... gawk
configure: error: Invalid value --with-l10n-base, /builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/obj-l10n/../../mozilla-esr10 doesn't exist
</pre>
Standard8 ported some build changes over to comm-esr10, and we're respinning.


==== Fennec ====
==Build 2==
===Preparing to start Automation===
<small>[[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|detailed instructions]]</small>
* clobber Any master, release-comm-esr10 branch, any builder
* Double land [http://hg.mozilla.org/build/buildbot-configs/rev/d63bb0b59645 config bump] and [http://hg.mozilla.org/build/buildbot-configs/rev/d63bb0b59645 l10n mozconfig fix].
* Retag buildbot-configs, buildbotcustom, & tools with build & release tags. <small>''([[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|details]])''</small>
** buildbot-configs: hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD2}
** buildbotcustom and tools, on build1 revision: hg tag THUNDERBIRD_10_0_5esr_BUILD2
* update and reconfigure the master
* dry run automation start
<pre>
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
    --branch comm-esr10 --build-number 2 \
    --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
    --dryrun -l localhost:9001
</pre>
* start automation ''&larr; monitor progress on buildbot (e.g. [http://buildbot-master34.build.mozilla.org:8001/waterfall?num_events=10&category=release-comm-esr10- comm-esr10])''
** the same command as above but without the "--dryrun"


===== Fennec repacks =====
=== E-mail Metrics ===
N/A for build 2


=== Unittests / Talos ===
=== E-mail mirrors@mozilla.org ===
<small>[[Release:Release_Automation_on_Mercurial:Documentation#Talos_.26_Unittests|details]]</small>
N/A
# Look to see that they ran
# Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
# Make sure there's no red/failures that we need to have a dev look at


=== XULRunner ===
=== Edit rsync exclude files ===
N/A


====Source====
=== Tag ===
No problems.
=== Bouncer Submitter ===
N/A
=== Source ===
Failed to clobber, {{bug|760389}}. Manually clobbered and rebuilt, green.


====Build====
Standard8 discovered it had the wrong mozilla-esr10 code - {{bug|760397}}.


=== Partner Repack ===
=== Build ===
==== Firefox ====
Also didn't clobber. Standard8 discovered that they have a gecko version of 10.0.6esrpre, so are getting source from mozilla-esr10's default branch - {{bug|760390}}
==== Fennec ====


=== Android Signing ===
Another respin required.
Instructions in [https://intranet.mozilla.org/Build:MobileSigning MobileSigning]


Now run on signing1.
==Build 3==
=== L10N verify ===
===Preparing to start Automation===
<small>[[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|detailed instructions]]</small>
* clobber Any master, release-comm-esr10 branch, any builder
* do manual clobbers of source, *_build, *repack*
* Double land [http://hg.mozilla.org/build/buildbotcustom/rev/83fd5c0b5818 source builder fix] and [http://hg.mozilla.org/build/buildbot-configs/rev/0aebc3c5abcc config bump]
* Standard8 landed the [http://hg.mozilla.org/releases/comm-esr10/rev/939269a86b88 mozconfig fix]
* Retag buildbot-configs, buildbotcustom, & tools with build & release tags <small>''([[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|details]])''</small>
** buildbotcustom & buildbot-configs: hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD3}
** tools, on build2 revision: hg tag -r 82e252ae7d68 THUNDERBIRD_10_0_5esr_BUILD3
* update and reconfigure the master
* dry run automation start
<pre>
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
    --branch comm-esr10 --build-number 3 \
    --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
    --dryrun -l localhost:9001
2012-06-01 03:51:57,136 : INFO : Branch comm-esr10 is enabled on master for thunderbird
2012-06-01 03:51:57,636 : INFO : Comparing thunderbird mozconfigs to nightly mozconfigs...
2012-06-01 03:51:58,037 : ERROR : found in mail/config/mozconfigs/macosx-universal/release but not in mail/config/mozconfigs/macosx-universal/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,037 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in macosx64 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --with-macbundlename-prefix=Firefox', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --disable-install-strip']!
2012-06-01 03:51:58,038 : ERROR : found in mail/config/mozconfigs/macosx-universal/nightly but not in mail/config/mozconfigs/macosx-universal/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:58,469 : ERROR : found in mail/config/mozconfigs/win32/release but not in mail/config/mozconfigs/win32/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,469 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']!
2012-06-01 03:51:58,470 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:58,754 : ERROR : found in mail/config/mozconfigs/linux64/release but not in mail/config/mozconfigs/linux64/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,754 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in linux64 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --with-ccache=/usr/bin/ccache', 'export MOZILLA_OFFICIAL=1', 'export MOZ_TELEMETRY_REPORTING=1', "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'"]!
2012-06-01 03:51:58,754 : ERROR : found in mail/config/mozconfigs/linux64/nightly but not in mail/config/mozconfigs/linux64/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:59,058 : ERROR : found in mail/config/mozconfigs/linux32/release but not in mail/config/mozconfigs/linux32/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:59,059 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in linux ['ac_add_options --enable-update-channel=nightly', 'CXX=$REAL_CXX', 'CXX="ccache $REAL_CXX"', 'CC="ccache $REAL_CC"', 'mk_add_options PROFILE_GEN_SCRIPT=@TOPSRCDIR@/build/profile_pageloader.pl', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --with-ccache=/usr/bin/ccache', 'export MOZILLA_OFFICIAL=1', 'export MOZ_TELEMETRY_REPORTING=1', "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'"]!
2012-06-01 03:51:59,059 : ERROR : found in mail/config/mozconfigs/linux32/nightly but not in mail/config/mozconfigs/linux32/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:59,059 : ERROR : Error verifying mozconfigs
</pre>
Which is fallout from the mozconfig change
* start automation wth -b to bypass error ''&larr; monitor progress on buildbot (e.g. [http://buildbot-master34.build.mozilla.org:8001/waterfall?num_events=10&category=release-comm-esr10- comm-esr10])''
** the same command as above but without the "--dryrun"


=== E-mail Metrics ===
N/A for build 3
=== E-mail mirrors@mozilla.org ===
N/A


=== Checksums ===
=== Edit rsync exclude files ===
N/A


=== Updates ===
=== Tag ===
No problems
=== Bouncer Submitter ===
N/A
=== Source ===
No problems


==== Update verify ====
=== Build ===
==== en-US ====
* linux32 ok
* linux64 and Mac failed in the compile step:
<pre>abort: unknown revision 'THUNDERBIRD_10_0_5esr_RELEASE'!
Time: real 0.900 secs (user 0.467+0.000 sys 0.100+0.000)
Traceback (most recent call last):
  File "/builds/slave/tb-rel-comm-esr10-osx64-bld/build/client.py", line 618, in <module>
Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-esr10/', '/builds/slave/tb-rel-comm-esr10-osx64-bld/build/mozilla']
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_10_0_5esr_RELEASE', '-R', '/builds/slave/tb-rel-comm-esr10-osx64-bld/build/mozilla', '--verbose', '--time']
</pre>
which is related to {{bug|757798}}. Solution is to manually pull in /builds/hg-shared/releases/mozilla-esr10 on bld-centos5-64-vmw-003 and moz2-darwin10-slave44, then clobber the build dirs. Also do the pull on other slaves that have repo to avoid later (potential) bustage in l10n repacks). The rebuild linux64 and mac en-US (green)
* win32 failed on the same share issue; updated the share & manual clobber, then rebuild (green)


=== Major Update ===
==== l10n repacks ====
Major update generation is not kicked off automatically. You should only create them once the final builds for the "latest" build are ready.
No problems
For instance, the 3.6.26 builds got created a week before the 10.0 release but the "final" 10.0 builds were not created after few days.


Follow instructions from the [[Release:Release_Automation_on_Mercurial:Documentation#Major_Update_.28Firefox_only.29|Major update]] documentation.
=== Checksums ===
No problems


==== Major Update verify ====
=== Updates ===
No problems
==== Update verify ====
No problems


=== Reset reserved slaves ===
=== Reset reserved slaves ===
No done, other releases using machines/master.


=== Check permissions / AV scan ===
=== Check permissions / AV scan ===
It is supposed to be triggered automatically after updates are done.
No problems
File a bug if it doesn't. <small>[[Release:Release_Automation_on_Mercurial:Documentation#Antivirus_check|details]]</small>


=== Push to internal mirrors ===
=== Push files to internal mirrors ===
This is done by automation for rapid release betas. '''Note''': if you have to reconfig the release buildbot master in during the release, the uptake monitoring will fail ({{bug|629648}}). That means you'll need to manually send the "ready for releasetest" emails when you believe things are "good enough". Refer to a prior release for email details.
No problems
 
For other releases, this should be done manually at this point, after check permissions / AV scan have completed.
* Make sure the rsync excludes have been updated ([[Releases/BuildNotesTemplate#Edit_rsync_exclude_files]])
* Force build the push to mirrors builder. Make sure at least 15 minutes have elapsed since editing the rysnc module to give it time to sync out.
 
[https://intranet.mozilla.org/ReleaseEngineering/Release/Primer#Mirrors_.28internal_.26_external.29 Mirrors Policy]


=== Final verification ===
=== Final verification ===
Record receipt of "completed_final_verification" emails (1 per platform)
No problems


=== Going to Beta ===
'''{{Bug|760807}} will means we need to redo win32'''
<font color='orange'>NOTE: Skip to "Before Release Day" for '''beta''' releases</font>


==== Run backupsnip ====
== Build 3 Spin 2 ==
NOTE: Remember to do this at least '''2 hours''' ahead of the expected "go to beta" email. <small>''[[Release:Release_Automation_on_Mercurial:Documentation#Prepare_for_Beta_Release_.28Firefox_only.29|details]]''</small>
===Preparing to start Automation===
* rather than update the release config, move the tags etc etc etc when we are rebuilding windows only we do this file munging on ftp (hard linking):
# tbirdbld@stage
/pub/mozilla.org/thunderbird/candidates/10.0.5esr-candidates
cp -alv build3{,.pre-win32-rebuild}/
cd build3
rm -rf jsshell-win32.zip *SUMS* update/win32/ win32/ win32_info.txt
ie hard link as a quick backup, then remove win32 specific files and SUMS files that will need be regenerated
* manual clobber of tb-rel-comm-esr10* on mw32-ix-slave02 and 03


Remember to ssh to aus3-staging using your short LDAP name and then run <code>sudo su - ffxbld</code> once on.  If you intend to use screen, initiate your screen session while you are still yourself.
=== E-mail Metrics ===
N/A


==== Push to beta ====
=== E-mail mirrors@mozilla.org ===
Wait for "go" from release managers<br />
N/A
For Firefox:
* Publish (and record run time here): <small>''[[Release:Release_Automation_on_Mercurial:Documentation#Beta_Release_.28Firefox_only.29|details]]''</small>
* on completion, email r-d that VERSION updates are live on the beta channel


For Android:
=== Edit rsync exclude files ===
* publish on Google Play (nee Android Marketplace), using [[#Publish_Fennec_to_the_Android_Market|instructions below]]
N/A
* on completions, email r-d that VERSION is live on Google Play


Wait for confirmation from QA that each of the above releases is okay, then close the tracking ticket.
=== Tag ===
N/A
=== Bouncer Submitter ===
N/A
=== Source ===
N/A


=== Before Release Day ===
=== Build ===
<!-- not for beta -->
==== en-US ====
Assuming this is not a chemspill.
Used the rebuild button, no problems
==== l10n repacks ====
mw32-ix-slave10 didn't have makensis.exe renamed to makensisu-2.46.exe, so repack 6 was using 2.33 when configure ran. Job aborted, clobbered, nsis fixed, rebooted; sr removed from candidate dir; builder rebuilt.


==== Run backupsnip for release snippets ====
All green after that.
Do this the night before release day as it can take up to two hours.


=== Push index file to mirrors ===
=== Checksums ===
Rebuilt. Failed at the very end where it tried to recreate the contrib dirs which already existed. Harmless so ignored.


=== Push to external mirrors ===
=== Updates ===
If this is a chemspill you will want to ping justdave (or other sysadmin) in #release-drivers and ask them to enable CDN.
Backed out update verify configs, and rebuilt. No problems.


* Remove previously added blacklisting entry from stage.mozilla.org:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude
==== Update verify ====
* Replace old version with the current one in stage.mozilla.org:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
No problems


See [https://intranet.mozilla.org/RelEngWiki/index.php/Managing_Rsync_Modules details]
=== Check permissions / AV scan ===
Both rebuilt. Permissions orange with:
WARN: target directory /pub/mozilla.org/thunderbird/releases/10.0.5esr/ exists
which is fine because that dir exists but is empty after emptying the earlier builds out.


=== Release Day ===
A/V no problems.
==== Publish Fennec to the Play Store (for beta releases and release releases) ====
=====Push the files (for beta releases and release releases)=====
* Use [http://hg.mozilla.org/build/braindump/file/tip/releases-related/push_fennec.sh this script] and run it as ffxbld@stage (please push you changes prior to using it, and grab latest on stage via <tt>curl -O http://hg.mozilla.org/build/braindump/raw-file/tip/releases-related/push_fennec.sh</tt>)


=====Play Store (for beta releases and release releases)=====
=== Push files to internal mirrors ===
'''IMPORTANT -''' Make sure to follow instructions!
Failed because thunderbird/releases/10.0.5esr already existed. Removed this and rebuilt. Twice it turns out, but the first one succeeded ok.
* download multi apk (e.g. [http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/10.0b6-candidates/build1/android/multi/fennec-10.0b6.multi.android-arm.apk multi/fennec-10.0b6.multi.android-arm.apk])
* visit https://market.android.com/publish
* choose "Firefox Beta" or "Firefox"
* select the "APK Files" tab and choose "Upload APK"
** choose the multi apk that you downloaded and hit "upload"
*** NOTE: the progress bar does not seem to show progress (at least on Google Chrome)
** when it finishes uploading, verify VersionCode is the build date (almost anyway, the market reports an hour later than our buildID)
** hit "Save"
* activate new apk (You should see an Error message since both apk's are active)
* deactivate old apk (You should not see the Error message anymore)
* hit "Save"
* If this is a beta 1, go to "Product Details" tab -> Recent Changes, change the url to  http://www.mozilla.com/en-US/mobile/{VERSION}beta/releasenotes/
** and hit "Save"


====Push XULRunner to Mirrors ====
=== Final verification ===
All green, but not actually checking the releasetest channel, so not verifying bouncer links OK - {{bug|761951}}. While writing the patch for that tested that win32 works OK on releasetest.


==== Update XULRunner wiki page ====
=== Backup snippets ===
For major releases or chemspills, update the links on:
time ~/bin/backupsnip Thunderbird-10.0.5esr-build3
* https://developer.mozilla.org/en/XULRunner
real 0m37.623s
* https://developer.mozilla.org/en/Gecko_SDK


Also for older releases:
=== Reset reserved slaves ===
* https://developer.mozilla.org/en/XULRunner_1.9.2_Release_Notes
In this case it would be about evacuating the slaves from bm34, which is limited to T'bird releases again, but we'll need them for 14.0b1 anyway.
* https://developer.mozilla.org/en/Gecko_SDK


==== Check Throttling  ====
=== Push to esr channel ===
See http://update-watch.localgho.st/release/ for example AUS links
time ~/bin/pushsnip Thunderbird-10.0.5esr-build3


Some links to check:
=== Update symlinks ===
* Automatic (idle time check) update links:
<pre>
** 10.0.2 en-US [https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml linux64]
#tbirdbld@stage
** 9.0 en-US [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86_64-gcc3/en-US/release/update.xml linux64]
cd /pub/mozilla.org/thunderbird/releases
** 10.0.2esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/WINNT_x86-msvc/en-US/esr/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86-gcc3/en-US/esr/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86_64-gcc3/en-US/esr/update.xml linux64]
rm latest-10.0esr; ln -s 10.0.5esr latest-10.0esr
** 10.0esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/WINNT_x86-msvc/en-US/esr/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86-gcc3/en-US/esr/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86_64-gcc3/en-US/esr/update.xml linux64]
</pre>
** 3.6.27 en-US [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Darwin_x86-gcc3-u-ppc-i386/en-US/release/update.xml mac]
* Manual update links:
** 10.0.2 en-US [https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml?force=1 linux64]
** 9.0 en-US [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86_64-gcc3/en-US/release/update.xml?force=1 linux64]
** 10.0.2esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/WINNT_x86-msvc/en-US/esr/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86-gcc3/en-US/esr/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86_64-gcc3/en-US/esr/update.xml?force=1 linux64]
** 10.0esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/WINNT_x86-msvc/en-US/esr/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86-gcc3/en-US/esr/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86_64-gcc3/en-US/esr/update.xml?force=1 linux64]
** 3.6.27 en-US [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Darwin_x86-gcc3-u-ppc-i386/en-US/release/update.xml?force=1 mac]
 
==== Push to Release Channel (for beta releases and release releases) ====
Once there is enough uptake and we get "go" from release driver.
 
==== Update symlinks ====
Once we're signed off on the release channel.
===== Remove index.html files =====
If you created them earlier to hide the release.
 
==== Update browser choice website ====

Latest revision as of 23:50, 11 June 2012

Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

  • bug 760353 - make source-package fails on comm-esr10 because MKDIR is undefined
  • bug 760361 - l10n repacks failing in configure
  • bug 760385 - we clobber hg shares more frequently than necessary dupe of bug 748158
  • bug 760389 - some builders not clobbered on build2
  • bug 760390 - en-US builds not pulling right mozilla-esr10 revision
  • bug 760397 - source builder not getting right mozilla-esr10 revision
  • bug 760807 - Program Compatibility Assistant shown on Windows 7 when setting as default mail client (NSIS version)
  • bug 761951 - not checking releasetest in final verification
  • bug 763769 - release sanity complains about mozconfig differences

Build Engineers

nthomas - Tracking bug: bug 744599

Signed-off Revision(s)

L10N changesets: unchanged

Build1:

Build2:

Build 3:

Build data

Firefox

Build # Type Build ID Build machine Time to build
1 Linux 20120531193048 mv-moz2-linux-ix-slave09 38 mins, 47 secs
Linux64 bld-centos5-64-vmw-003 45 mins, 40 secs
Mac moz2-darwin10-slave44 3 hrs, 32 mins, 31 secs
Windows mw32-ix-slave26 1 hrs, 43 mins, 28 secs
2 Linux 20120601013708 mv-moz2-linux-ix-slave09 38 mins, 13 secs
3 Linux 20120601041309 mv-moz2-linux-ix-slave09 37 mins, 26 secs
Linux64 bld-centos5-64-vmw-003 41 mins, 53 secs
Mac moz2-darwin10-slave44 3 hrs, 32 mins, 5 secs
Windows mw32-ix-slave02 1 hrs, 39 mins, 32 secs
3 Spin 2 Windows 20120605154056 mw32-ix-slave02 1 hrs, 39 mins, 18 secs

Notes

Build 1

Preparing to start Automation

detailed instructions

  • first TB ESR release on Firefox infra so nothing to clobber
  • Reserve slaves (you need ~5 each of linux, linux64, win32 (w32), and macosx64 (moz2-darwin10)). Doing this 2-3 hours in advance should speed things up. e.g.:
[cltbld@buildbot-master34 master]$ pwd
/builds/buildbot/build1/master
[cltbld@buildbot-master43 master]$ cat reserved_slaves
6
  • For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)
# tbirdbld@stage
cd /pub/mozilla.org/thunderbird/nightly/
version=10.0.5esr
mkdir ../candidates/$version-candidates
ln -s ../candidates/$version-candidates $version-candidates
  • No l10n changes
  • Double land automation configs
  • Tag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD1}
  • update and reconfigure the master
  • dry run automation start
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
    --branch comm-esr10 --build-number 1 \
    --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
    --dryrun -l localhost:9001
  • start automation ← monitor progress on buildbot (e.g. comm-esr10)
    • the same command as above but without the "--dryrun"

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org". Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on #metrics. 

Thunderbird,thunderbird,thunderbird,10.0.5,10.0
Thunderbird,thunderbird,thunderbird,10.0.5esr,10.0

E-mail mirrors@mozilla.org

N/A

Edit rsync exclude files

N/A

Tag

No problems

Bouncer Submitter

No problems

Source

Failed: 

make source-package hg-bundle HG_BUNDLE_REVISION=7c765c397d25
 in dir /builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb (timeout 2700 secs)
make[1]: Entering directory `/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb/mail/installer'
Packaging source tarball...
p ../../mozilla/dist/source/
make[1]: p: Command not found
make[1]: [source-package] Error 127 (ignored)
(cd /builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10 && gtar -c --owner=0 --group=0 --numeric-owner --mode="go-w" --exclude=".hg*" --exclude="CVS" --exclude=".cvs*" --exclude=".mozconfig*" --exclude="*.pyc" --exclude="/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/mozilla/Makefile" --exclude="/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/mozilla/dist" --exclude="objdir-tb" -f - ../comm-esr10) | bzip2 -vf > ../../mozilla/dist/source/thunderbird-10.0.5esr.source.tar.bz2
/bin/sh: ../../mozilla/dist/source/thunderbird-10.0.5esr.source.tar.bz2: No such file or directory
gtar: Removing leading `../' from member names
make[1]: *** [source-package] Error 1
make[1]: Leaving directory `/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb/mail/installer'
make: *** [source-package] Error 2

bug 760353 filed.

Build

en-US

No problems.

Repacks

Failing in configure, bug 760361 filed on: 

/builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/configure
Adding configure options from /builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/.mozconfig:
  --with-l10n-base=/builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/obj-l10n/../../mozilla-esr10
  --enable-application=mail
  --enable-official-branding
  --enable-update-channel=esr
  --enable-update-packaging
creating cache ./config.cache
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking build system type... x86_64-unknown-linux-gnu
checking for gawk... gawk
configure: error: Invalid value --with-l10n-base, /builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/obj-l10n/../../mozilla-esr10 doesn't exist

Standard8 ported some build changes over to comm-esr10, and we're respinning.

Build 2

Preparing to start Automation

detailed instructions

  • clobber Any master, release-comm-esr10 branch, any builder
  • Double land config bump and l10n mozconfig fix.
  • Retag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
    • buildbot-configs: hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD2}
    • buildbotcustom and tools, on build1 revision: hg tag THUNDERBIRD_10_0_5esr_BUILD2
  • update and reconfigure the master
  • dry run automation start
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
    --branch comm-esr10 --build-number 2 \
    --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
    --dryrun -l localhost:9001
  • start automation ← monitor progress on buildbot (e.g. comm-esr10)
    • the same command as above but without the "--dryrun"

E-mail Metrics

N/A for build 2

E-mail mirrors@mozilla.org

N/A

Edit rsync exclude files

N/A

Tag

No problems.

Bouncer Submitter

N/A

Source

Failed to clobber, bug 760389. Manually clobbered and rebuilt, green.

Standard8 discovered it had the wrong mozilla-esr10 code - bug 760397.

Build

Also didn't clobber. Standard8 discovered that they have a gecko version of 10.0.6esrpre, so are getting source from mozilla-esr10's default branch - bug 760390

Another respin required.

Build 3

Preparing to start Automation

detailed instructions

  • clobber Any master, release-comm-esr10 branch, any builder
  • do manual clobbers of source, *_build, *repack*
  • Double land source builder fix and config bump
  • Standard8 landed the mozconfig fix
  • Retag buildbot-configs, buildbotcustom, & tools with build & release tags (details)
    • buildbotcustom & buildbot-configs: hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD3}
    • tools, on build2 revision: hg tag -r 82e252ae7d68 THUNDERBIRD_10_0_5esr_BUILD3
  • update and reconfigure the master
  • dry run automation start
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
    --branch comm-esr10 --build-number 3 \
    --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
    --dryrun -l localhost:9001
2012-06-01 03:51:57,136 : INFO : Branch comm-esr10 is enabled on master for thunderbird
2012-06-01 03:51:57,636 : INFO : Comparing thunderbird mozconfigs to nightly mozconfigs...
2012-06-01 03:51:58,037 : ERROR : found in mail/config/mozconfigs/macosx-universal/release but not in mail/config/mozconfigs/macosx-universal/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,037 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in macosx64 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --with-macbundlename-prefix=Firefox', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --disable-install-strip']!
2012-06-01 03:51:58,038 : ERROR : found in mail/config/mozconfigs/macosx-universal/nightly but not in mail/config/mozconfigs/macosx-universal/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:58,469 : ERROR : found in mail/config/mozconfigs/win32/release but not in mail/config/mozconfigs/win32/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,469 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']!
2012-06-01 03:51:58,470 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:58,754 : ERROR : found in mail/config/mozconfigs/linux64/release but not in mail/config/mozconfigs/linux64/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,754 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in linux64 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --with-ccache=/usr/bin/ccache', 'export MOZILLA_OFFICIAL=1', 'export MOZ_TELEMETRY_REPORTING=1', "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'"]!
2012-06-01 03:51:58,754 : ERROR : found in mail/config/mozconfigs/linux64/nightly but not in mail/config/mozconfigs/linux64/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:59,058 : ERROR : found in mail/config/mozconfigs/linux32/release but not in mail/config/mozconfigs/linux32/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:59,059 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in linux ['ac_add_options --enable-update-channel=nightly', 'CXX=$REAL_CXX', 'CXX="ccache $REAL_CXX"', 'CC="ccache $REAL_CC"', 'mk_add_options PROFILE_GEN_SCRIPT=@TOPSRCDIR@/build/profile_pageloader.pl', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --with-ccache=/usr/bin/ccache', 'export MOZILLA_OFFICIAL=1', 'export MOZ_TELEMETRY_REPORTING=1', "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'"]!
2012-06-01 03:51:59,059 : ERROR : found in mail/config/mozconfigs/linux32/nightly but not in mail/config/mozconfigs/linux32/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:59,059 : ERROR : Error verifying mozconfigs

Which is fallout from the mozconfig change

  • start automation wth -b to bypass error ← monitor progress on buildbot (e.g. comm-esr10)
    • the same command as above but without the "--dryrun"

E-mail Metrics

N/A for build 3

E-mail mirrors@mozilla.org

N/A

Edit rsync exclude files

N/A

Tag

No problems

Bouncer Submitter

N/A

Source

No problems

Build

en-US

  • linux32 ok
  • linux64 and Mac failed in the compile step:
abort: unknown revision 'THUNDERBIRD_10_0_5esr_RELEASE'!
Time: real 0.900 secs (user 0.467+0.000 sys 0.100+0.000)
Traceback (most recent call last):
  File "/builds/slave/tb-rel-comm-esr10-osx64-bld/build/client.py", line 618, in <module>
Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-esr10/', '/builds/slave/tb-rel-comm-esr10-osx64-bld/build/mozilla']
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_10_0_5esr_RELEASE', '-R', '/builds/slave/tb-rel-comm-esr10-osx64-bld/build/mozilla', '--verbose', '--time']

which is related to bug 757798. Solution is to manually pull in /builds/hg-shared/releases/mozilla-esr10 on bld-centos5-64-vmw-003 and moz2-darwin10-slave44, then clobber the build dirs. Also do the pull on other slaves that have repo to avoid later (potential) bustage in l10n repacks). The rebuild linux64 and mac en-US (green)

  • win32 failed on the same share issue; updated the share & manual clobber, then rebuild (green)

l10n repacks

No problems

Checksums

No problems

Updates

No problems

Update verify

No problems

Reset reserved slaves

No done, other releases using machines/master.

Check permissions / AV scan

No problems

Push files to internal mirrors

No problems

Final verification

No problems

bug 760807 will means we need to redo win32

Build 3 Spin 2

Preparing to start Automation

  • rather than update the release config, move the tags etc etc etc when we are rebuilding windows only we do this file munging on ftp (hard linking):
# tbirdbld@stage
/pub/mozilla.org/thunderbird/candidates/10.0.5esr-candidates
cp -alv build3{,.pre-win32-rebuild}/
cd build3
rm -rf jsshell-win32.zip *SUMS* update/win32/ win32/ win32_info.txt

ie hard link as a quick backup, then remove win32 specific files and SUMS files that will need be regenerated

  • manual clobber of tb-rel-comm-esr10* on mw32-ix-slave02 and 03

E-mail Metrics

N/A

E-mail mirrors@mozilla.org

N/A

Edit rsync exclude files

N/A

Tag

N/A

Bouncer Submitter

N/A

Source

N/A

Build

en-US

Used the rebuild button, no problems

l10n repacks

mw32-ix-slave10 didn't have makensis.exe renamed to makensisu-2.46.exe, so repack 6 was using 2.33 when configure ran. Job aborted, clobbered, nsis fixed, rebooted; sr removed from candidate dir; builder rebuilt.

All green after that.

Checksums

Rebuilt. Failed at the very end where it tried to recreate the contrib dirs which already existed. Harmless so ignored.

Updates

Backed out update verify configs, and rebuilt. No problems.

Update verify

No problems

Check permissions / AV scan

Both rebuilt. Permissions orange with: 

WARN: target directory /pub/mozilla.org/thunderbird/releases/10.0.5esr/ exists

which is fine because that dir exists but is empty after emptying the earlier builds out.

A/V no problems.

Push files to internal mirrors

Failed because thunderbird/releases/10.0.5esr already existed. Removed this and rebuilt. Twice it turns out, but the first one succeeded ok.

Final verification

All green, but not actually checking the releasetest channel, so not verifying bouncer links OK - bug 761951. While writing the patch for that tested that win32 works OK on releasetest.

Backup snippets

time ~/bin/backupsnip Thunderbird-10.0.5esr-build3
real	0m37.623s

Reset reserved slaves

In this case it would be about evacuating the slaves from bm34, which is limited to T'bird releases again, but we'll need them for 14.0b1 anyway.

Push to esr channel

time ~/bin/pushsnip Thunderbird-10.0.5esr-build3

Update symlinks

#tbirdbld@stage
cd /pub/mozilla.org/thunderbird/releases
rm latest-10.0esr; ln -s 10.0.5esr latest-10.0esr
Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Thunderbird_10.0.5esr/BuildNotes&oldid=440140"