Releases/Thunderbird 10.0.5esr/BuildNotes: Difference between revisions

(Created page with "=Notes About Releasing= Please update the [https://wiki.mozilla.org/index.php?title=Releases/BuildNotesTemplate Notes Template] and the [https://intranet.mozilla.org/Build:Releas...")
 
 
(34 intermediate revisions by 2 users not shown)
Line 4: Line 4:
= Bugs hit =
= Bugs hit =
Enter any bugs pre-existing or newly discovered and filed during the release:
Enter any bugs pre-existing or newly discovered and filed during the release:
* {{bug|}} - description
* {{bug|760353}} - make source-package fails on comm-esr10 because MKDIR is undefined
*
* {{bug|760361}} - l10n repacks failing in configure
* <strike>{{bug|760385}} - we clobber hg shares more frequently than necessary</strike> dupe of {{bug|748158}}
* {{bug|760389}} - some builders not clobbered on build2
* {{bug|760390}} - en-US builds not pulling right mozilla-esr10 revision
* {{bug|760397}} - source builder not getting right mozilla-esr10 revision
* {{bug|760807}} - Program Compatibility Assistant shown on Windows 7 when setting as default mail client (NSIS version)
* {{bug|761951}} - not checking releasetest in final verification
* {{bug|763769}} - release sanity complains about mozconfig differences


=Build Engineers=
=Build Engineers=
Line 11: Line 18:


=Signed-off Revision(s)=
=Signed-off Revision(s)=
* comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/570c22432c13
 
* mozilla-esr10: http://hg.mozilla.org/releases/mozilla-esr10/rev/X
L10N changesets: unchanged
* L10N changesets: unchanged
 
Build1:
* comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/570c22432c13 on default
* mozilla-esr10: http://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH
 
Build2:
* comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/48677c4f453a on GECKO1005_2012053119_RELBRANCH
* mozilla-esr10: http://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH
 
Build 3:
* comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/939269a86b88 on GECKO1005_2012053119_RELBRANCH
* mozilla-esr10: https://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH


=Build data=
=Build data=
Line 26: Line 44:
| rowspan="4" | 1  
| rowspan="4" | 1  
| Linux
| Linux
| rowspan="4" |  
| rowspan="4" | 20120531193048
|  
| mv-moz2-linux-ix-slave09
|  
| 38 mins, 47 secs
|-
| Linux64
| bld-centos5-64-vmw-003
| 45 mins, 40 secs
|-
| Mac
| moz2-darwin10-slave44
| 3 hrs, 32 mins, 31 secs
|-
| Windows
| mw32-ix-slave26
| 1 hrs, 43 mins, 28 secs
|-
| rowspan="1" | 2
| Linux
| rowspan="1" | 20120601013708
| mv-moz2-linux-ix-slave09
| 38 mins, 13 secs
|-
| rowspan="4" | 3
| Linux
| rowspan="4" | 20120601041309
| mv-moz2-linux-ix-slave09
| 37 mins, 26 secs
|-
|-
| Linux64
| Linux64
|  
| bld-centos5-64-vmw-003
|  
| 41 mins, 53 secs
|-
|-
| Mac
| Mac
|  
| moz2-darwin10-slave44
|  
| 3 hrs, 32 mins, 5 secs
|-
|-
| Windows  
| Windows  
|  
| mw32-ix-slave02
|  
| 1 hrs, 39 mins, 32 secs
|-
| 3 Spin 2
| Windows
| 20120605154056
| mw32-ix-slave02
| 1 hrs, 39 mins, 18 secs
|}
|}


Line 62: Line 110:
  ln -s ../candidates/$version-candidates $version-candidates
  ln -s ../candidates/$version-candidates $version-candidates
* No l10n changes
* No l10n changes
<font color='red'>'''DONE TO HERE'''</font>
* Double land automation configs
* Double land automation configs
* Tag buildbot-configs, buildbotcustom, & tools with build & release tags. <small>''([[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|details]])''</small>
* Tag buildbot-configs, buildbotcustom, & tools with build & release tags. <small>''([[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|details]])''</small>
Line 71: Line 118:
cd /builds/buildbot/build1/master
cd /builds/buildbot/build1/master
source ../bin/activate
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr --branch releases/comm-esr10 --build-number 1 \
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
    --branch comm-esr10 --build-number 1 \
     --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
     --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
     --dryrun -l localhost:9001
     --dryrun -l localhost:9001
Line 77: Line 125:
* start automation ''&larr; monitor progress on buildbot (e.g. [http://buildbot-master34.build.mozilla.org:8001/waterfall?num_events=10&category=release-comm-esr10- comm-esr10])''
* start automation ''&larr; monitor progress on buildbot (e.g. [http://buildbot-master34.build.mozilla.org:8001/waterfall?num_events=10&category=release-comm-esr10- comm-esr10])''
** the same command as above but without the "--dryrun"
** the same command as above but without the "--dryrun"
<font color='red'>'''NOTES TRIMMED TO HERE'''</font>


=== E-mail Metrics ===
=== E-mail Metrics ===
Use the address "metrics-alerts < AT > mozilla < PERIOD > org". <small><em>Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on <tt>#metrics</tt>.</em></small>
Use the address "metrics-alerts < AT > mozilla < PERIOD > org". <small><em>Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on <tt>#metrics</tt>.</em></small>
Thunderbird,thunderbird,thunderbird,10.0.5,10.0
Thunderbird,thunderbird,thunderbird,10.0.5esr,10.0


For 3.6.25 you would send
=== E-mail mirrors@mozilla.org ===
Firefox,firefox,firefox,3.6.25,3.6
N/A
Firefox,firefox,firefox,3.6.26pre,3.6


For non-3.6 releases, we need to include Fennec in the email if we build Fennec.
=== Edit rsync exclude files ===
N/A


For 9.0b2 it would be
=== Tag ===
Firefox,firefox,firefox,9.0b2,9.0
No problems
Fennec,firefox-mobile,fennec,9.0b2,9.0


And for 9.0 final it's
=== Bouncer Submitter ===
Firefox,firefox,firefox,9.0,9.0
No problems
Fennec,firefox-mobile,fennec,9.0,9.0


ESRs look like this:
=== Source ===
  Firefox,firefox,firefox,10.0.3,10.0
Failed:
Firefox,firefox,firefox,10.0.3esr,10.0
<pre>
make source-package hg-bundle HG_BUNDLE_REVISION=7c765c397d25
  in dir /builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb (timeout 2700 secs)
make[1]: Entering directory `/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb/mail/installer'
Packaging source tarball...
p ../../mozilla/dist/source/
make[1]: p: Command not found
make[1]: [source-package] Error 127 (ignored)
(cd /builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10 && gtar -c --owner=0 --group=0 --numeric-owner --mode="go-w" --exclude=".hg*" --exclude="CVS" --exclude=".cvs*" --exclude=".mozconfig*" --exclude="*.pyc" --exclude="/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/mozilla/Makefile" --exclude="/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/mozilla/dist" --exclude="objdir-tb" -f - ../comm-esr10) | bzip2 -vf > ../../mozilla/dist/source/thunderbird-10.0.5esr.source.tar.bz2
/bin/sh: ../../mozilla/dist/source/thunderbird-10.0.5esr.source.tar.bz2: No such file or directory
gtar: Removing leading `../' from member names
make[1]: *** [source-package] Error 1
make[1]: Leaving directory `/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb/mail/installer'
make: *** [source-package] Error 2
</pre>
{{Bug|760353}} filed.


=== E-mail mirrors@mozilla.org ===
=== Build ===
Not necessary for rapid release betas, files only go as far as the 'internal mirrors'.
==== en-US ====
No problems.


Coordinate yourself with the person creating the ESR and the latest release which normally go out at the same time.  
==== Repacks ====
Failing in configure, {{bug|760361}} filed on:
<pre>/builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/configure
Adding configure options from /builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/.mozconfig:
  --with-l10n-base=/builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/obj-l10n/../../mozilla-esr10
  --enable-application=mail
  --enable-official-branding
  --enable-update-channel=esr
  --enable-update-packaging
creating cache ./config.cache
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking build system type... x86_64-unknown-linux-gnu
checking for gawk... gawk
configure: error: Invalid value --with-l10n-base, /builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/obj-l10n/../../mozilla-esr10 doesn't exist
</pre>
Standard8 ported some build changes over to comm-esr10, and we're respinning.


They want to know approximately what time push to external mirrors will occur. Ideally this is a 24hr notice to them, with the estimated time.
==Build 2==
===Preparing to start Automation===
<small>[[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|detailed instructions]]</small>
* clobber Any master, release-comm-esr10 branch, any builder
* Double land [http://hg.mozilla.org/build/buildbot-configs/rev/d63bb0b59645 config bump] and [http://hg.mozilla.org/build/buildbot-configs/rev/d63bb0b59645 l10n mozconfig fix].
* Retag buildbot-configs, buildbotcustom, & tools with build & release tags. <small>''([[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|details]])''</small>
** buildbot-configs: hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD2}
** buildbotcustom and tools, on build1 revision: hg tag THUNDERBIRD_10_0_5esr_BUILD2
* update and reconfigure the master
* dry run automation start
<pre>
<pre>
Subject: Firefox {VERSION1} [and {VERSION2}] coming to mirrors on {DATE}
cd /builds/buildbot/build1/master
Body:
source ../bin/activate
We're planning on pushing our Firefox {VERSION1} [and {VERSION2}] release(s) to mirrors 24 hours before {DATE}.
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
# If this is a chemspill, please mention that in the email as it will mean pinging
    --branch comm-esr10 --build-number 2 \
# for CDN to be enabled when the push to internal mirrors happens
    --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
    --dryrun -l localhost:9001
</pre>
* start automation ''&larr; monitor progress on buildbot (e.g. [http://buildbot-master34.build.mozilla.org:8001/waterfall?num_events=10&category=release-comm-esr10- comm-esr10])''
** the same command as above but without the "--dryrun"
 
=== E-mail Metrics ===
N/A for build 2


If you believe these releases or these dates will cause any problems,
=== E-mail mirrors@mozilla.org ===
please notify release at mozilla.com. If you have any problems with mirror
N/A
status/weights/etc, please notify mirror-submissions at mozilla.org.
</pre>


=== Edit rsync exclude files ===
=== Edit rsync exclude files ===
Edit /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude on stage so that it excludes the current release. This prevents external mirrors from picking up the new release prematurely.
N/A
 
For rapid release betas, this is not necessary as there's already an exclude that matches beta releases.


=== Tag ===
=== Tag ===
 
No problems.
=== Bouncer Submitter ===
=== Bouncer Submitter ===
 
N/A
 
=== Source ===
=== Source ===
Failed to clobber, {{bug|760389}}. Manually clobbered and rebuilt, green.


Standard8 discovered it had the wrong mozilla-esr10 code - {{bug|760397}}.


=== Build ===
=== Build ===
==== Firefox ====
Also didn't clobber. Standard8 discovered that they have a gecko version of 10.0.6esrpre, so are getting source from mozilla-esr10's default branch - {{bug|760390}}


===== Firefox repacks =====
Another respin required.


==== Fennec ====
==Build 3==
===Preparing to start Automation===
<small>[[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|detailed instructions]]</small>
* clobber Any master, release-comm-esr10 branch, any builder
* do manual clobbers of source, *_build, *repack*
* Double land [http://hg.mozilla.org/build/buildbotcustom/rev/83fd5c0b5818 source builder fix] and [http://hg.mozilla.org/build/buildbot-configs/rev/0aebc3c5abcc config bump]
* Standard8 landed the [http://hg.mozilla.org/releases/comm-esr10/rev/939269a86b88 mozconfig fix]
* Retag buildbot-configs, buildbotcustom, & tools with build & release tags <small>''([[Release:Release_Automation_on_Mercurial:Documentation#Starting_the_automation|details]])''</small>
** buildbotcustom & buildbot-configs: hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD3}
** tools, on build2 revision: hg tag -r 82e252ae7d68 THUNDERBIRD_10_0_5esr_BUILD3
* update and reconfigure the master
* dry run automation start
<pre>
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
    --branch comm-esr10 --build-number 3 \
    --release-config release-thunderbird-comm-esr10.py --products thunderbird  \
    --dryrun -l localhost:9001
2012-06-01 03:51:57,136 : INFO : Branch comm-esr10 is enabled on master for thunderbird
2012-06-01 03:51:57,636 : INFO : Comparing thunderbird mozconfigs to nightly mozconfigs...
2012-06-01 03:51:58,037 : ERROR : found in mail/config/mozconfigs/macosx-universal/release but not in mail/config/mozconfigs/macosx-universal/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,037 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in macosx64 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --with-macbundlename-prefix=Firefox', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --disable-install-strip']!
2012-06-01 03:51:58,038 : ERROR : found in mail/config/mozconfigs/macosx-universal/nightly but not in mail/config/mozconfigs/macosx-universal/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:58,469 : ERROR : found in mail/config/mozconfigs/win32/release but not in mail/config/mozconfigs/win32/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,469 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']!
2012-06-01 03:51:58,470 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:58,754 : ERROR : found in mail/config/mozconfigs/linux64/release but not in mail/config/mozconfigs/linux64/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,754 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in linux64 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --with-ccache=/usr/bin/ccache', 'export MOZILLA_OFFICIAL=1', 'export MOZ_TELEMETRY_REPORTING=1', "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'"]!
2012-06-01 03:51:58,754 : ERROR : found in mail/config/mozconfigs/linux64/nightly but not in mail/config/mozconfigs/linux64/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:59,058 : ERROR : found in mail/config/mozconfigs/linux32/release but not in mail/config/mozconfigs/linux32/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:59,059 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in linux ['ac_add_options --enable-update-channel=nightly', 'CXX=$REAL_CXX', 'CXX="ccache $REAL_CXX"', 'CC="ccache $REAL_CC"', 'mk_add_options PROFILE_GEN_SCRIPT=@TOPSRCDIR@/build/profile_pageloader.pl', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --with-ccache=/usr/bin/ccache', 'export MOZILLA_OFFICIAL=1', 'export MOZ_TELEMETRY_REPORTING=1', "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'"]!
2012-06-01 03:51:59,059 : ERROR : found in mail/config/mozconfigs/linux32/nightly but not in mail/config/mozconfigs/linux32/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:59,059 : ERROR : Error verifying mozconfigs
</pre>
Which is fallout from the mozconfig change
* start automation wth -b to bypass error ''&larr; monitor progress on buildbot (e.g. [http://buildbot-master34.build.mozilla.org:8001/waterfall?num_events=10&category=release-comm-esr10- comm-esr10])''
** the same command as above but without the "--dryrun"


===== Fennec repacks =====
=== E-mail Metrics ===
N/A for build 3


=== Unittests / Talos ===
=== E-mail mirrors@mozilla.org ===
<small>[[Release:Release_Automation_on_Mercurial:Documentation#Talos_.26_Unittests|details]]</small>
N/A
# Look to see that they ran
# Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
# Make sure there's no red/failures that we need to have a dev look at


=== XULRunner ===
=== Edit rsync exclude files ===
N/A


====Source====
=== Tag ===
No problems
=== Bouncer Submitter ===
N/A
=== Source ===
No problems


====Build====
=== Build ===
 
==== en-US ====  
=== Partner Repack ===
* linux32 ok
==== Firefox ====
* linux64 and Mac failed in the compile step:
==== Fennec ====
<pre>abort: unknown revision 'THUNDERBIRD_10_0_5esr_RELEASE'!
 
Time: real 0.900 secs (user 0.467+0.000 sys 0.100+0.000)
=== Android Signing ===
Traceback (most recent call last):
Instructions in [https://intranet.mozilla.org/Build:MobileSigning MobileSigning]
  File "/builds/slave/tb-rel-comm-esr10-osx64-bld/build/client.py", line 618, in <module>
 
Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-esr10/', '/builds/slave/tb-rel-comm-esr10-osx64-bld/build/mozilla']
Now run on signing1.
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_10_0_5esr_RELEASE', '-R', '/builds/slave/tb-rel-comm-esr10-osx64-bld/build/mozilla', '--verbose', '--time']
=== L10N verify ===
</pre>
which is related to {{bug|757798}}. Solution is to manually pull in /builds/hg-shared/releases/mozilla-esr10 on bld-centos5-64-vmw-003 and moz2-darwin10-slave44, then clobber the build dirs. Also do the pull on other slaves that have repo to avoid later (potential) bustage in l10n repacks). The rebuild linux64 and mac en-US (green)
* win32 failed on the same share issue; updated the share & manual clobber, then rebuild (green)


==== l10n repacks ====
No problems


=== Checksums ===
=== Checksums ===
No problems


=== Updates ===
=== Updates ===
 
No problems
==== Update verify ====
==== Update verify ====
 
No problems
=== Major Update ===
Major update generation is not kicked off automatically. You should only create them once the final builds for the "latest" build are ready.
For instance, the 3.6.26 builds got created a week before the 10.0 release but the "final" 10.0 builds were not created after few days.
 
Follow instructions from the [[Release:Release_Automation_on_Mercurial:Documentation#Major_Update_.28Firefox_only.29|Major update]] documentation.
 
==== Major Update verify ====


=== Reset reserved slaves ===
=== Reset reserved slaves ===
No done, other releases using machines/master.


=== Check permissions / AV scan ===
=== Check permissions / AV scan ===
It is supposed to be triggered automatically after updates are done.
No problems
File a bug if it doesn't. <small>[[Release:Release_Automation_on_Mercurial:Documentation#Antivirus_check|details]]</small>
 
=== Push to internal mirrors ===
This is done by automation for rapid release betas. '''Note''': if you have to reconfig the release buildbot master in during the release, the uptake monitoring will fail ({{bug|629648}}). That means you'll need to manually send the "ready for releasetest" emails when you believe things are "good enough". Refer to a prior release for email details.
 
For other releases, this should be done manually at this point, after check permissions / AV scan have completed.
* Make sure the rsync excludes have been updated ([[Releases/BuildNotesTemplate#Edit_rsync_exclude_files]])
* Force build the push to mirrors builder. Make sure at least 15 minutes have elapsed since editing the rysnc module to give it time to sync out.


[https://intranet.mozilla.org/ReleaseEngineering/Release/Primer#Mirrors_.28internal_.26_external.29 Mirrors Policy]
=== Push files to internal mirrors ===
No problems


=== Final verification ===
=== Final verification ===
Record receipt of "completed_final_verification" emails (1 per platform)
No problems


=== Going to Beta ===
'''{{Bug|760807}} will means we need to redo win32'''
<font color='orange'>NOTE: Skip to "Before Release Day" for '''beta''' releases</font>


==== Run backupsnip ====
== Build 3 Spin 2 ==
NOTE: Remember to do this at least '''2 hours''' ahead of the expected "go to beta" email. <small>''[[Release:Release_Automation_on_Mercurial:Documentation#Prepare_for_Beta_Release_.28Firefox_only.29|details]]''</small>
===Preparing to start Automation===
* rather than update the release config, move the tags etc etc etc when we are rebuilding windows only we do this file munging on ftp (hard linking):
# tbirdbld@stage
/pub/mozilla.org/thunderbird/candidates/10.0.5esr-candidates
cp -alv build3{,.pre-win32-rebuild}/
cd build3
rm -rf jsshell-win32.zip *SUMS* update/win32/ win32/ win32_info.txt
ie hard link as a quick backup, then remove win32 specific files and SUMS files that will need be regenerated
* manual clobber of tb-rel-comm-esr10* on mw32-ix-slave02 and 03


Remember to ssh to aus3-staging using your short LDAP name and then run <code>sudo su - ffxbld</code> once on.  If you intend to use screen, initiate your screen session while you are still yourself.
=== E-mail Metrics ===
N/A


==== Push to beta ====
=== E-mail mirrors@mozilla.org ===
Wait for "go" from release managers<br />
N/A
For Firefox:
* Publish (and record run time here): <small>''[[Release:Release_Automation_on_Mercurial:Documentation#Beta_Release_.28Firefox_only.29|details]]''</small>
* on completion, email r-d that VERSION updates are live on the beta channel


For Android:
=== Edit rsync exclude files ===
* publish on Google Play (nee Android Marketplace), using [[#Publish_Fennec_to_the_Android_Market|instructions below]]
N/A
* on completions, email r-d that VERSION is live on Google Play


Wait for confirmation from QA that each of the above releases is okay, then close the tracking ticket.
=== Tag ===
N/A
=== Bouncer Submitter ===
N/A
=== Source ===
N/A


=== Before Release Day ===
=== Build ===
<!-- not for beta -->
==== en-US ====
Assuming this is not a chemspill.
Used the rebuild button, no problems
==== l10n repacks ====
mw32-ix-slave10 didn't have makensis.exe renamed to makensisu-2.46.exe, so repack 6 was using 2.33 when configure ran. Job aborted, clobbered, nsis fixed, rebooted; sr removed from candidate dir; builder rebuilt.


==== Run backupsnip for release snippets ====
All green after that.
Do this the night before release day as it can take up to two hours.


=== Push index file to mirrors ===
=== Checksums ===
Rebuilt. Failed at the very end where it tried to recreate the contrib dirs which already existed. Harmless so ignored.


=== Push to external mirrors ===
=== Updates ===
If this is a chemspill you will want to ping justdave (or other sysadmin) in #release-drivers and ask them to enable CDN.
Backed out update verify configs, and rebuilt. No problems.


* Remove previously added blacklisting entry from stage.mozilla.org:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude
==== Update verify ====
* Replace old version with the current one in stage.mozilla.org:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
No problems


See [https://intranet.mozilla.org/RelEngWiki/index.php/Managing_Rsync_Modules details]
=== Check permissions / AV scan ===
Both rebuilt. Permissions orange with:
WARN: target directory /pub/mozilla.org/thunderbird/releases/10.0.5esr/ exists
which is fine because that dir exists but is empty after emptying the earlier builds out.


=== Release Day ===
A/V no problems.
==== Publish Fennec to the Play Store (for beta releases and release releases) ====
=====Push the files (for beta releases and release releases)=====
* Use [http://hg.mozilla.org/build/braindump/file/tip/releases-related/push_fennec.sh this script] and run it as ffxbld@stage (please push you changes prior to using it, and grab latest on stage via <tt>curl -O http://hg.mozilla.org/build/braindump/raw-file/tip/releases-related/push_fennec.sh</tt>)


=====Play Store (for beta releases and release releases)=====
=== Push files to internal mirrors ===
'''IMPORTANT -''' Make sure to follow instructions!
Failed because thunderbird/releases/10.0.5esr already existed. Removed this and rebuilt. Twice it turns out, but the first one succeeded ok.
* download multi apk (e.g. [http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/10.0b6-candidates/build1/android/multi/fennec-10.0b6.multi.android-arm.apk multi/fennec-10.0b6.multi.android-arm.apk])
* visit https://market.android.com/publish
* choose "Firefox Beta" or "Firefox"
* select the "APK Files" tab and choose "Upload APK"
** choose the multi apk that you downloaded and hit "upload"
*** NOTE: the progress bar does not seem to show progress (at least on Google Chrome)
** when it finishes uploading, verify VersionCode is the build date (almost anyway, the market reports an hour later than our buildID)
** hit "Save"
* activate new apk (You should see an Error message since both apk's are active)
* deactivate old apk (You should not see the Error message anymore)
* hit "Save"
* If this is a beta 1, go to "Product Details" tab -> Recent Changes, change the url to  http://www.mozilla.com/en-US/mobile/{VERSION}beta/releasenotes/
** and hit "Save"


====Push XULRunner to Mirrors ====
=== Final verification ===
All green, but not actually checking the releasetest channel, so not verifying bouncer links OK - {{bug|761951}}. While writing the patch for that tested that win32 works OK on releasetest.


==== Update XULRunner wiki page ====
=== Backup snippets ===
For major releases or chemspills, update the links on:
time ~/bin/backupsnip Thunderbird-10.0.5esr-build3
* https://developer.mozilla.org/en/XULRunner
real 0m37.623s
* https://developer.mozilla.org/en/Gecko_SDK


Also for older releases:
=== Reset reserved slaves ===
* https://developer.mozilla.org/en/XULRunner_1.9.2_Release_Notes
In this case it would be about evacuating the slaves from bm34, which is limited to T'bird releases again, but we'll need them for 14.0b1 anyway.
* https://developer.mozilla.org/en/Gecko_SDK


==== Check Throttling  ====
=== Push to esr channel ===
See http://update-watch.localgho.st/release/ for example AUS links
time ~/bin/pushsnip Thunderbird-10.0.5esr-build3


Some links to check:
=== Update symlinks ===
* Automatic (idle time check) update links:
<pre>
** 10.0.2 en-US [https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml linux64]
#tbirdbld@stage
** 9.0 en-US [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86_64-gcc3/en-US/release/update.xml linux64]
cd /pub/mozilla.org/thunderbird/releases
** 10.0.2esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/WINNT_x86-msvc/en-US/esr/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86-gcc3/en-US/esr/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86_64-gcc3/en-US/esr/update.xml linux64]
rm latest-10.0esr; ln -s 10.0.5esr latest-10.0esr
** 10.0esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/WINNT_x86-msvc/en-US/esr/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86-gcc3/en-US/esr/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86_64-gcc3/en-US/esr/update.xml linux64]
</pre>
** 3.6.27 en-US [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Darwin_x86-gcc3-u-ppc-i386/en-US/release/update.xml mac]
* Manual update links:
** 10.0.2 en-US [https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml?force=1 linux64]
** 9.0 en-US [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86_64-gcc3/en-US/release/update.xml?force=1 linux64]
** 10.0.2esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/WINNT_x86-msvc/en-US/esr/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86-gcc3/en-US/esr/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86_64-gcc3/en-US/esr/update.xml?force=1 linux64]
** 10.0esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/WINNT_x86-msvc/en-US/esr/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86-gcc3/en-US/esr/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86_64-gcc3/en-US/esr/update.xml?force=1 linux64]
** 3.6.27 en-US [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Darwin_x86-gcc3-u-ppc-i386/en-US/release/update.xml?force=1 mac]
 
==== Push to Release Channel (for beta releases and release releases) ====
Once there is enough uptake and we get "go" from release driver.
 
==== Update symlinks ====
Once we're signed off on the release channel.
===== Remove index.html files =====
If you created them earlier to hide the release.
 
==== Update browser choice website ====
Confirmed users
4,293

edits