Releases/Firefox 3.5.2/Post Mortem
This is the agenda and notes from the post mortem for Firefox 3.0.13 and 3.5.2 that took place on August 6 at 1pm PDT.
Please add your issues to the relevant section below.
- MFSAs online ahead of time? Put them on mozilla.com in the release notes as well?
- It would have been nice if we could have started builds prior to Wednesday afternoon. How can we keep from having embargoed bugs in the future?
- 3.0.13 was ready for beta early Sat morning. Decision did not come until later.
- waited 10pm friday until 4pm saturday
- Really great collaboration with some of the developers in bug verification
- Tripped on l10n makefile change. needed to redo l10n repacks/signing/updates.
- New signing worked nicely; needed it for all the re-signing
- This time, we pulled the 3.0.12 -> 3.5.1 major update before shipping at ss' request. Why? Should we continue to do that?
- rationale could be shared pre or post fact in release-drivers [juanb]
- did xulrunner builds at same time
- didnt do partner builds automatically
- Make sure both product & marketing teams are clear (and agree) on content objectives for what's new page.
- Denial of service on ocsp.globalsign.net caused by everyone trying to validate our EV cert on addons.mozilla.org (bug 508408)
- Several anti-virus products detected the Windows 3.0.13 and 3.5.2 installers as infected (false positive) (bug 508012)
- in general, development went okay
- need to work on advisories getting out earlier / on time
- some confusion because of the expediency of this release
- get bsterne mozilla.org check-in access (bsterne; ss will vouch)
- starting builds because we were under embargo hurt us
- in this case, closed source won because they could check in without anyone knowing
- need to work with security researchers to make this not an option on the table (dveditz/lucas)
- lots of great help from all over QA including the community
- a set of trusted QA community helped test this release so we could ship fast
- all around good QA work
- developers also helped a bunch with verifications
- need better idea of driver hand off
- QA was ready to go to bed with 3.0.13 on Friday night
- no decision until Saturday afternoon
- wasn't clear which driver to ping to get it done (ss)
- l10n makefile change messed up repacks
- need to get such changes better tested before they land
- automated tests for such changes should be required (ss)
- faster signing worked great
- had to resign because of the l10n issues
- went *really* fast, which was great
- pulling major update snippets at the last minute wasn't good
- manual task, prone to possible mistakes
- wasn't clear why it was happening
- needs to get communicated better what's happening and why via r-d
- wasn't communicated post-release either
- a follow-up discussion should occur on this (ss)
- XULRunner builds happened at the same time
- partner builds didn't happen at the same time
- automation for partner builds not ready yet
- hoping to have ready soon (joduinn)
- pages we pushed this time were more-or-less fine
- some work being done in the future to improve the messaging (jslater, beltzner)
- be sure to get sign off for these after changes are made
- kubla was slow, like usual
- loading the page with so many locales takes too long
- follow up bugs to file and push on
- need to get bugs fixed and pushed on (ss)
- DoS against EV provider caused AMO to "go down" for Firefox users only
- DoS likely caused by us and our users...
- working with GlobalSign to ensure they're ready for our users
- clearly wasn't the case this time, but we're following up (mrz)
- several antivirus vendors claimed we had a virus (false positive)
- caused by a change in compression for installers
- part of new signing process
- Tomcat followed up with vendors to get fixes out
- worked okay this time, but need to notify them next time
- maybe not take these changes during firedrill releases