Releases/Thunderbird 34.0b1/BuildNotes: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎issues: for hg slow on l10n)
(completed post release)
 
(10 intermediate revisions by 4 users not shown)
Line 3: Line 3:
* <s>(TB Drivers) [[Release:Release_Automation_on_Mercurial:Preparation#L10N_Changesets | Ship the l10n milestone]]
* <s>(TB Drivers) [[Release:Release_Automation_on_Mercurial:Preparation#L10N_Changesets | Ship the l10n milestone]]
* (TB Drivers) [[Release:Release_Automation_on_Mercurial:Starting_a_Release#Submit_to_Ship_It | Submit to Ship It]]</s>
* (TB Drivers) [[Release:Release_Automation_on_Mercurial:Starting_a_Release#Submit_to_Ship_It | Submit to Ship It]]</s>
* [[Release:Release_Automation_on_Mercurial:Updates#Push_snippets | Run pushsnip]]
* abandoned here
* [[Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Publish_in_Balrog | Publish in Balrog ]]
* [[Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Post-release_tasks | Post-release tasks]]


== issues ==
== issues ==
Line 16: Line 14:
*<s> (TB Drivers) [[Release:Release_Automation_on_Mercurial:Preparation#L10N_Changesets | Ship the l10n milestone]]
*<s> (TB Drivers) [[Release:Release_Automation_on_Mercurial:Preparation#L10N_Changesets | Ship the l10n milestone]]
* (TB Drivers) [[Release:Release_Automation_on_Mercurial:Starting_a_Release#Submit_to_Ship_It | Submit to Ship It]]</s>
* (TB Drivers) [[Release:Release_Automation_on_Mercurial:Starting_a_Release#Submit_to_Ship_It | Submit to Ship It]]</s>
* [[Release:Release_Automation_on_Mercurial:Updates#Push_snippets | Run pushsnip]]
* We are going to have a Build3
* [[Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Publish_in_Balrog | Publish in Balrog ]]
 
* [[Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Post-release_tasks | Post-release tasks]]
== issues ==
== issues ==
* linux compile failed, hg timeout
* linux compile failed, hg timeout
Line 38: Line 35:
** [http://hg.mozilla.org/build/tools/rev/72eb2a1ea2a1 Extra fix] landed, retag tools and rebuild all 40 l10n repack jobs
** [http://hg.mozilla.org/build/tools/rev/72eb2a1ea2a1 Extra fix] landed, retag tools and rebuild all 40 l10n repack jobs


* further l10n issues due to hg slowness: linux64 8/10, mac 10/10: 40 min timeout without output in configure, looks like it was cloning. Rebuild
* further l10n issues due to hg slowness: linux64 8/10, mac 8,10/10, win8,10/10: 40 min timeout without output in configure, looks like it was cloning. Rebuild
 
 
= Build 3 =
== Checklist ==
*<s> (TB Drivers) [[Release:Release_Automation_on_Mercurial:Preparation#L10N_Changesets | Ship the l10n milestone]]
* (TB Drivers) [[Release:Release_Automation_on_Mercurial:Starting_a_Release#Submit_to_Ship_It | Submit to Ship It]]</s>
* <s>[[Release:Release_Automation_on_Mercurial:Updates#Push_snippets | Run pushsnip]]</s>
* <s>[[Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Publish_in_Balrog | Publish in Balrog ]]
* [[Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Post-release_tasks | Post-release tasks]]</s>
 
== issues ==
* Two failed repacks:
** repack_8/10 failed for Thunderbird 34.0b1 build3 on linux
** repack_10/10 failed for Thunderbird 34.0b1 build3 on linux
because they were taken by a slave with disk space issues. Slave disabled and job re-triggered
* macosx64 build step failed for Thunderbird 34.0b1 build3 ([http://buildbot-master82.srv.releng.scl3.mozilla.com:8001/builders/release-comm-beta-macosx64_build/builds/4/steps/make_buildsymbols/logs/stdio%20 full log here])
<pre>
adding: XUL/122DCC0758BB3DB9839D90765AF6D39E0/XUL.sym
  zip warning:  file size changed while zipping XUL/122DCC0758BB3DB9839D90765AF6D39E0/XUL.sym
    (stored 0%)
  adding: XUL/122DCC0758BB3DB9839D90765AF6D39E0/XUL.tar.bz2
  zip I/O error: Result too large
  zip error: Output file write failure (write error on zip file)
  make: *** [buildsymbols] Error 14
  program finished with exit code 2
  elapsedTime=1830.229363
</pre>
This job was taken by bld-lion-r5-080 which is not in great shape. Slave disabled and job re-triggered.
* failed at check_permissions but actually it's a warning about the existence of /pub/mozilla.org/thunderbird/releases/34.0b1 on stage.mozilla.org
* failed at thunderbird_push_to_mirrors
<pre>rm -rf /pub/mozilla.org/thunderbird/releases/34.0b1</pre>
then run check permissions and push to mirrors again
* final verification falied: created [https://bugzilla.mozilla.org/show_bug.cgi?id=1099048 bug 1099048] to purge CDNs
** it took over an hour for the purge to "take" and final verify to pass. Three re-runs were required.
 
== Hold updates after accidental push ==
Updates were pushed after QA signed off (rather than waiting for an explicit go, which is what we do for Thunderbird). To fix, I updated the rule matching product: Thunderbird, channel: beta to have a mapping of Thunderbird-33.0b1-build2. Done via https://aus4-admin.mozilla.org/rules.html. When we're ready to go for real, the update shipping builder can be triggered again.
 
Republished after go from Standard8.

Latest revision as of 15:45, 18 November 2014

Build 1

Checklist

issues

hgtool and TB l10n repacks don't go well together :/ We need either fix hgtool or disable it. (rail's email) I killed all builds running. There will be build2 with hgtool disabled (rail).

Build 2

Checklist

issues

  • linux compile failed, hg timeout
  • repacks are failing
    • a lot of hg timeouts, buildbot kills the process
    • when hg does not time out, we have:
CalledProcessError: Command '['perl', '../../../../mozilla/tools/update-packaging/unwrap_full_update.pl', '/builds/slave/tb-rel-c-beta-l64_rpk_9-000000/comm-beta/obj-l10n/dist/update/linux-x86_64/sl/thunderbird-34.0b1.complete.mar']' returned non-zero exit status 2

changes introduced by Bug 1087104 http://hg.mozilla.org/releases/mozilla-beta/rev/ee2c3cfb4a7b, here ?? it worked fine in 33.0b1

CWD is different, in 33.0b1 was:

cwd: /builds/slave/tb-rel-c-beta-l64_rpk_5-000000/comm-beta/obj-l10n/mozilla/dist/current

in 34.0b1 is:

cwd: /builds/slave/tb-rel-c-beta-l64_rpk_9-000000/comm-beta/obj-l10n/dist/current
    • Standard8 landed a fix, rail retagged and rebuilt the jobs, but it still had problem
      Can't open perl script "../../../../mozilla/tools/update-packaging/unwrap_full_update.pl": No such file or directory
    • Extra fix landed, retag tools and rebuild all 40 l10n repack jobs
  • further l10n issues due to hg slowness: linux64 8/10, mac 8,10/10, win8,10/10: 40 min timeout without output in configure, looks like it was cloning. Rebuild


Build 3

Checklist

issues

  • Two failed repacks:
    • repack_8/10 failed for Thunderbird 34.0b1 build3 on linux
    • repack_10/10 failed for Thunderbird 34.0b1 build3 on linux

because they were taken by a slave with disk space issues. Slave disabled and job re-triggered

  • macosx64 build step failed for Thunderbird 34.0b1 build3 (full log here)
 adding: XUL/122DCC0758BB3DB9839D90765AF6D39E0/XUL.sym
   zip warning:  file size changed while zipping XUL/122DCC0758BB3DB9839D90765AF6D39E0/XUL.sym
     (stored 0%)
   adding: XUL/122DCC0758BB3DB9839D90765AF6D39E0/XUL.tar.bz2
   zip I/O error: Result too large
   zip error: Output file write failure (write error on zip file)
   make: *** [buildsymbols] Error 14
   program finished with exit code 2
   elapsedTime=1830.229363

This job was taken by bld-lion-r5-080 which is not in great shape. Slave disabled and job re-triggered.

  • failed at check_permissions but actually it's a warning about the existence of /pub/mozilla.org/thunderbird/releases/34.0b1 on stage.mozilla.org
  • failed at thunderbird_push_to_mirrors
rm -rf /pub/mozilla.org/thunderbird/releases/34.0b1

then run check permissions and push to mirrors again

  • final verification falied: created bug 1099048 to purge CDNs
    • it took over an hour for the purge to "take" and final verify to pass. Three re-runs were required.

Hold updates after accidental push

Updates were pushed after QA signed off (rather than waiting for an explicit go, which is what we do for Thunderbird). To fix, I updated the rule matching product: Thunderbird, channel: beta to have a mapping of Thunderbird-33.0b1-build2. Done via https://aus4-admin.mozilla.org/rules.html. When we're ready to go for real, the update shipping builder can be triggered again.

Republished after go from Standard8.