Releases/Firefox 3.6/Post Mortem

Call in details

• Mozilla Mountain View (The Bridge)
• 650-903-0800 or 650-215-1282 x92 Conf# 8605 (US/INTL)
• 1-800-707-2533 (pin 369) Conf# 8605 (US)
• irc.mozilla.org #planning for backchannel

Agenda

Key question: how can we do this even better next time?

How did Firefox 3.6 measure up to the goals for the release?

• faster development cycle (less than 1 yr for a meaningful product)
  • done...6.5 months
• deliver everything required so mobile team could ship their product the way they wanted
  • done
• improve startup time and other perceivable ways of tracking speed
  • definitely faster than 3.5, but didn't really hit goal on startup time. "Perception of performance" goals were a mixed bag.
• other comments:
  • define release criteria & goals more clearly up front
  • PR messaging around faster development cycle gets tricky - reporters start writing stories about a delayed release once it slips past initial expectations, so let's change the story to be more about why we're doing the faster releases (better for users, better for the web, etc)
  • nebulousness around ship date (may have) caused extra work/confusion for add-ons team...ex. Personas integration w/AMO site

What was good about the delivery schedule?

• props to PR efforts for rolling with the schedule changes and being ready at all times no matter what
  • and props in return from the PR team for really quick turnarounds & approvals on top priority issues
• rapid beta cycles worked very well - makes every part of the change management process easier
  • props to Build & QA teams for helping make this happen
• good work building up beta testing community very quickly (using tactics like Google snippets, placements on key mozilla.com pages, community marketing, etc)
• general consensus that having more betas & quick beta cycles was good, and we should consider changing the naming structure for next time ("beta version 1, 2, 3, etc" vs "beta 1, beta 2, beta 3, etc")
  • one potential drawback is that this could affect adoption among users (if they don't sense that betas are really changing)
• lack of changes on the front end made it easier to have a rapid beta cycle
• PR: held dev news/moz blog posts until the best time for worldwide impact rather than pushing them out right away (ex. waiting until Monday morning to announce rather than doing it late on a Friday night)
• AMO compatibility issues didn't cause major release problems

What caused delays in the delivery schedule?

• late-breaking file API changes slowed things down
• new heightened focus on Crash/Kill issues pushed back the release date
  • at the very least, it took a lot of engineering attention...results were good, but it impacted ability to ship on time
  • improvements made by the Crash/Kill process were worth the extra month of dev time before the release, though (next time let's start focusing on this at the beginning of the cycle rather than 3/4 in)
  • the increased number of blockers caused by Crash/Kill created a perception that the release was further away...also invited scope creep
• FFTs before we ship take a long time...amounts to something like 5% of the overall ship schedule
  • we should take a look at the last round of manual verifications and figure out how we can condense that
  • focus more on the incremental changes per beta
• adding features later in the process causes churn in QA - having to write new test cases, etc - and distracts them from the work of shipping betas, etc
• tegra project took away QA resources...should have asked for time estimate up front rather than just assigning the work without a full understanding of impact
• still feels awkward and surprising that we're still finding sites that are broken with our JS engine even at beta 5 or RC (a big cause of respins late in the game)
  • some of this was b/c of changes later in the process (ex. beta 5) that caused the problems...need to be aware that these changes may have other repercussions (although most of the time the changes have to be made)
• missed some Facebook compatibility issues, but a lot of that was on their side

What infrastructure improvements helped

• automated testing, branch mechanics have helped
• having dirty profiles was incredible...helped us catch a lot of things
• TS testing very helpful
• lots of of ways of understanding the effects of code changes we were making as we were making them
• kept Firebug compatible throughout all beta cycles

What infrastructure improvements helped, and what could be improved?

• automated testing for add-ons
• would be great if there was a suite of Firebug tests we could put into the tree so we could understand when we're breaking Firebug (Rob has a patch for this that we could build on)

Effectiveness of PR and marketing efforts, and things that could be improved

How things went on "ship day", and what could be improved