Release Management/Rapid Betas

Benefits of Rapid Betas

• shorter verification turn-around time
• freed up RelEng/QA resources
• late-breaking beta issues can be much more easily resolved
• more user testing over the 6 week cycle

Proposed Timeline

• -4/23
  • Consider major pain points, begin scoping work.
• 4/24-6/4
  • Complete scoping work, begin implementation
  • Communicate the plan to Mozilla community
  • Move to Tuesday go, Wednesday ship? (see QA questions below)
  • Any necessary in-product changes should land while FF15 is on m-c
• 6/5-7/16
  • Complete implementation, begin process testing on Nightly/Aurora
• Week of 7/17
  • Push out our first rapid beta while FF15 on mozilla-beta

In-Product

Questions

• What do we do about non-admin users? What's the current experience? Can we back off on offering them an update by 7 days?
• Will the user be prompted at all to "apply the update now"? Can we disable that?
• If the user's browser downloads an update, then the user doesn't close and open their browser for 3 days, what build will be applied?
• Is the user going to be prompted every day to update, or will it be truely silent and the user won't notice it until the next day.
• Will our current beta users be happy with the additional bandwidth of potentially downloading updates every day?

Blockers

• Background updates will be landing in FF15.
  • Feature page
  • Tracking bug 307181

QA

Questions

• breakdown of when sign off has uncovered new regressions in a beta? (and if it has, was it through automation or manual (human) tests?)
  • If we find that sign-off has not found new critical beta regressions, can we start shipping betas on Wednesdays after automated testing runs through (prior to rapid betas being implemented)? Bug verification, exploratory testing, and other smoke tests could continue out-of-band.
• can automating the release test plan be done against nightly/aurora in preparation for moving to faster beta releases?
• Can we automate mobile update testing? If so, we can even make the weekly push to the Play Store automatic.

Blockers

• Update testing (and smoke tests) should be automated and worked into the build/push flow

Support

Questions

• any pain points in supporting Aurora? (expand those into to-do items for supporting rapid betas)

RelEng/IT

Questions

• any issues with automated pushsnip to beta channel?
  • Needs to be gated on QA. We already automatically push to betatest. Catlee 12:48, 13 April 2012 (PDT)
• can we add inputs from QA to incorporate the sign-off as part of automation?
• any possible issues with load (or mirrors) if we move to more frequent releases of beta on bouncer?
• Using the signing key appears to be a manual process currently, can this be automated?
  • Already is! (except for android) - Catlee 12:39, 13 April 2012 (PDT)
• Can we do the work associated with rapid betas be done in such a way that we could move back to weekly betas in the case of unforseen problems?
• Do we need to tag/package source for rapid betas?

Crash-Kill Team & Soccorro

Questions

• What are the implications of switching to rapid betas? How would crash analysis be different than m-c or m-a?
• Do we need to implement windowing over multiple build IDs to emulate having a large beta population on a single beta?
• What is a unit of 'enough' user testing data (days?) for getting feedback on a regression fix?

Blockers

Scoping happening here

L10N

Questions

• What do we do with l10n milestones?
• Could we work towards automatically pulling from a beta branch?
• Could l10n_changesets be dynamically created from the csets on beta branch for locales in shipped_locales?
• Can l10n sign offs be automated? Test suites that check for anything other than string changes?

Blockers

• Eliminating the need for l10n sign-offs (already so much churn for weekly beta, rapid betas would likely be untennable)

Release Management

Update - RelEng (and lsblakk) met and discussed the complete process of releasing betas and have scoped the work involved in meeting this project's goals here

Questions

• What should we call beta nightlies (Rapid Betas)?
• How do we communicate beta nightlies?
• The beta release process is a useful tool for verifying the final release process. If we change the final release process, how will we verify the changes or bring new people up to speed without affecting the timing of the final releases?
  • Do we need to build in a slightly larger window for the final release to allow for these types of issues?

Blockers

• Need to change the process of approving for Beta to first go through Aurora, for sake of risk mitigation

Product Marketing & PR

Update - Grace & Laura discussed; EJ added input; questions and blockers below.

Questions

• This will have an impact on how we could deploy a What's New Page which we've planning to do for sometime. We will likely still be able to, but we'll have to coordinate much closer on when and how those pages are turned off and on. What would be the details of getting a paged turned on or off?
• How will this effect release notes? Will we update them every day? Once a week?
• How will this effect how we track comments/feedback on Input?
• Will this effect how we track ADUs/ADIs with metrics tools?
• How will users be alerted to the updates? Will it cause update fatigue?
• Is this the new way of doing things or a short term thing we are testing?
• Does this mean new features can land in beta?
• Do we have to update our communications around what to expect for each channel?
• Will this change the release timeline?

Blockers

• Blog Post communicating change on FoF; must be reviewed by PMMs and PR
• Alignment on naming (we don't see a reason to change the name and didn't like adding "rapid" to releases)
• Figuring out how to make release notes work for all audiences/communications
• This will impact if/when/how often we want to communicate about betas
• Updating all channel comms