Release Management/Rapid Betas

Benefits of Rapid Betas

• shorter verification turn-around time
• freed up RelEng/QA resources
• late-breaking beta issues can be much more easily resolved
• more user testing of the latest bits over the 6 week cycle

Proposed Timeline

• -4/23
  • Consider major pain points, begin scoping work.
• 4/24-6/4
  • Complete scoping work, begin implementation
  • Communicate the plan to Mozilla community
  • Move to Tuesday go, Wednesday ship? (see QA questions below)
  • Any necessary in-product changes should land while FF15 is on m-c
• 6/5-7/16
  • Complete implementation, begin process testing on Nightly/Aurora
• Week of 7/17
  • Push out our first rapid beta while FF15 on mozilla-beta

In-Product

Questions

• What do we do about non-admin users? What's the current experience? Can we back off on offering them an update by 7 days?
  • should be possible be possible by only incrementing the update xml's appVersion attribute once a week see bug 753103
• Will the user be prompted at all to "apply the update now"? Can we disable that?
• If the user's browser downloads an update, then the user doesn't close and open their browser for 3 days, what build will be applied?
• Is the user going to be prompted every day to update, or will it be truely silent and the user won't notice it until the next day.
• Will our current beta users be happy with the additional bandwidth of potentially downloading updates every day?

Blockers

• Background updates will be landing in FF15.
  • Feature page
  • Tracking bug 307181, bug 753103
  • Telemetry bugs (for getting uptime data): bug 727184 - Increase granularity of telemetry uptime measurement, bug 733591 - Send uptime with telemetry data
  • Decreasing prompt level since download (if you don't close the browser): bug 754470 - Once we have rapid betas, we should change the beta channel to prompt users about a downloaded update once a week

Automation

Questions

• What are the communication points that need to happen (what needs to be communicated and when between the QA & RelEng)
• What can we do to keep automation running as smoothly as possible with known/unknown failures or automation glitches? ie: we will still need troubleshooting from QA when automation cuts out due to errors

QA

Questions

• breakdown of when sign off has uncovered new regressions in a beta? (and if it has, was it through automation or manual (human) tests?)
  • If we find that sign-off has not found new critical beta regressions, can we start shipping betas on Wednesdays after automated testing runs through (prior to rapid betas being implemented)? Bug verification, exploratory testing, and other smoke tests could continue out-of-band.
• can automating the release test plan be done against nightly/aurora in preparation for moving to faster beta releases?
• Can we automate mobile update testing? If so, we can even make the weekly push to the Play Store automatic.

Blockers

• Update testing (and smoke tests) should be automated and worked into the build/push flow

Support

Questions

• any pain points in supporting Aurora? (expand those into to-do items for supporting rapid betas)

RelEng/IT

Questions

• any issues with automated pushsnip to beta channel?
  • Needs to be gated on QA. We already automatically push to betatest. Catlee 12:48, 13 April 2012 (PDT)
• can we add inputs from QA to incorporate the sign-off as part of automation?
• any possible issues with load (or mirrors) if we move to more frequent releases of beta on bouncer?
• Using the signing key appears to be a manual process currently, can this be automated?
  • Already is! (except for android) - Catlee 12:39, 13 April 2012 (PDT)
  • note: upcoming osx10.8 signing is explicitly excluded
• Can we do the work associated with rapid betas be done in such a way that we could move back to weekly betas in the case of unforseen problems?
• Do we need to tag/package source for rapid betas?

Blockers

• Tracking bug for RelEng work is bug 747168. Most notable items on it are:
  • Automate android signing bug 705807
  • Move master-side logic to client-side scripts: bug 748773, bug 748794, bug 748796, bug 748798, bug 748800
  • Make mail notification work without a reconfig bug 749587
  • Automate snippet optimization bug 748801
  • Switch to using a different L10n url to get changesets for desktop: bug 751703, and keep the changeset info from each build in the manifest (to be created by releng as a replacement for tagging)

• unrelated but blocker: supporting new OS/platforms.

Crash-Kill Team & Soccorro

Questions

• What are the implications of switching to rapid betas? How would crash analysis be different than m-c or m-a?
• Do we need to implement windowing over multiple build IDs to emulate having a large beta population on a single beta?
• What is a unit of 'enough' user testing data (days?) for getting feedback on a regression fix?

Blockers

• Scoping happening here
• Tracking bug 752956
• Requirements
• Implementation Plan

L10N

Answers/Resolutions

• Keep sign-offs on release and beta channel, continue without signoffs on aurora and central
• Drop the use of milestones at least for rapid betas
• Get l10n-changesets from urls that give the current signed-off state for a product/version
  • These urls need to updated on migration day every six weeks
  • These urls are already existing, using the av= keyword argument with the right code instead of a ms= keyword

Questions

• What do we do with l10n milestones?
• Could we work towards automatically pulling from a beta branch? This doesn't really scale well.
• Could l10n_changesets be dynamically created from the csets on beta branch for locales in shipped_locales?
• Can l10n sign offs be automated? Test suites that check for anything other than string changes?

New questions:

• What happens with non-rapid betas, notably TB/SeaMonkey? (No change, continue as usual)
• Do we need a "sealed" status for release, or is that good to go without milestones, too? (That is up to the needs of the l10n drivers, do you want a tagged repo every 6 weeks when a release goes out?)

Blockers

• None, but a releng bug on using the new url: bug 751703

Release Management

Update - RelEng (and lsblakk) met and discussed the complete process of releasing betas and have scoped the work involved in meeting this project's goals here

Questions

• What should we call beta nightlies (Rapid Betas)?
• How do we communicate beta nightlies?
• The beta release process is a useful tool for verifying the final release process. If we change the final release process, how will we verify the changes or bring new people up to speed without affecting the timing of the final releases?
  • Do we need to build in a slightly larger window for the final release to allow for these types of issues?

Blockers

• Need to change the process of approving for Beta to first go through Aurora, for sake of risk mitigation
• We need to watch for more stretch across multiple versions of beta than we currently have across weekly betas
• Feature request: getting BuildIDs in Input to get more data on where feedback is coming from
• What's new page - either always on or always off but we'll need a better plan

Metrics

We did a sample analysis on the crash report data for beta for last 10 days of april. The 9 out of the top 10 crashers in the 1st hour remain in the top 10 of the last hour of the 10 day period. We probably will not be able to pick up enough reports to catch the rare crashes on day 1, so these will persist in subsequent builds and will be picked up later.

In summary: - Using Rapid Builds we will be able to push bug fixes faster. - Rare signatures will be picked up later (once enough reports come in) and in this regard Rapid Builds is not much different from weekly builds.

Product Marketing & PR

Update - Grace & Laura discussed; EJ added input; Laura discussed in channel meeting, wrote down answers.

Questions & Responses

• This will have an impact on how we could deploy a What's New Page which we've planning to do for sometime. We will likely still be able to, but we'll have to coordinate much closer on when and how those pages are turned off and on. What would be the details of getting a paged turned on or off?
  • No clear answer on this. This is not a blocker, but something to keep in mind for the future to find a solution.
• How will this effect release notes? Will we update them every day? Once a week?
  • Will keep doing release notes the way we do Aurora--stand them up with initial release and make minor changes as needed; don't think we will see major changes between first release and end of the cycle.
• How will this effect how we track comments/feedback on Input?
  • Yes, it will. Will need a way to correlate comments to buildIDs.
• Will this effect how we track ADUs/ADIs with metrics tools?
  • Yes, Alex is filing a bug.
• How will users be alerted to the updates? Will it cause update fatigue?
  • No because this will only be implemented after silent updates.
• Is this the new way of doing things or a short term thing we are testing?
  • Its the plan for the foreseeable future, but may be changed depending on resources, needs, etc.
• Does this mean new features can land in beta?
  • It makes it less risky to land features, but any feature additions will be discussed in depth before landing; don't think this will cause an increase in the landing of new features.
• Do we have to update our communications around what to expect for each channel?
  • I don't think it does, apart from a general blog post announcing the change.
• Will this change the release timeline?
  • No, this shouldn't effect release timeline at all.

Blockers & Answers

• Blog Post communicating change on FoF; must be reviewed by PMMs and PR
  • This will be done in conjunction with Alex Keybl, Mark LaVine/EJ and LMesa; timing tbd.
• Alignment on naming (we don't see a reason to change the name and didn't like adding "rapid" to releases)
  • Grace, LMesa and EJ to discuss and come back with proposals
    • continuous betas?
    • rolling betas?
    • seamless betas?
• Figuring out how to make release notes work for all audiences/communications
  • Don't think this will be a huge concern; see above.
• This will impact if/when/how often we want to communicate about betas
  • This is unlikely as the changes that will be done between betas will probably stay with bug fixes, security fixes or feature back outs.
• Updating all channel comms
  • Unlikely outside of blog post mentioned above. Ej/LM/ML to discuss.

Project Status

In-Product

Bug # or Issue	Status	ETA	Contact
Background updates	on track for landing in FF15	FF15	rstrong

Tracked bugs

Full Query

ID	Summary	Priority	Status
307181	Eliminate wait while restarting Firefox after update (apply update in background)	--	RESOLVED
727184	Increase granularity of telemetry uptime measurement	--	RESOLVED
733591	Send uptime with telemetry data	--	RESOLVED
753103	Only bump the update xml's appVersion attribute once a week	P3	RESOLVED
754470	Once we have rapid betas, we should change the beta channel to prompt users about a downloaded update once a week	--	NEW

5 Total; 1 Open (20%); 4 Resolved (80%); 0 Verified (0%);

Automation

Bug # or Issue	Status	ETA	Contact
No current issues	on track		ctalbert

QA

Bug # or Issue	Status	ETA	Contact
Update testing (and smoke tests) should be automated and worked into the build/push flow			ashughes

Support

Bug # or Issue	Status	ETA	Contact
No current issues	on track		Cheng

RelEng/IT

Bug # or Issue	Status	ETA	Contact
Implementing daily beta builds	blocked on dealing with new OS/platforms		Joduinn

Tracked bugs

Full Query

ID	Summary	Priority	Status
705807	Android signing-on-demand	P3	RESOLVED
747168	[tracking bug] release automation support for daily betas	P3	RESOLVED
748773	read revisions required for a release from a common file	P3	RESOLVED
748794	read en-US revision for release builds on the client side	P3	RESOLVED
748796	replace SingleSourceFactory with a mozharness script	P3	RESOLVED
748798	replace ReleaseUpdatesFactory with a simplified mozharness script	P3	RESOLVED
748800	replace TuxedoEntrySubmitterFactory with a mozharness script	P3	RESOLVED
748801	run snippet optimization as part of the release process	P3	RESOLVED
749587	use properties for things in release MailNotifiers that change every release	P3	RESOLVED
751703	release_sanity.py needs to check locales/locale repos from a different url	P3	RESOLVED

10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);

Crash-Kill Team & Soccorro

Bug # or Issue	Status	ETA	Contact
Implementation of Socorro handling for daily betas	project is scoped for implementation by 7/17	7/17	Smooney/Laura

Tracked bugs

Full Query

ID	Summary	Priority	Status
752956	[tracker] Add Rapid Beta support to Socorro	--	RESOLVED
755293	[TRACKER] UI changes to support rapid betas	P1	RESOLVED
755297	Database changes to support rapid betas	P1	RESOLVED
755301	Middleware changes to support rapid betas	P1	RESOLVED
755304	Audit FTP scraper cron job for rapid beta support	--	RESOLVED

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);

L10N

Bug # or Issue	Status	ETA	Contact
No current issues (RelEng has the bug on l10n changesets)	on track		Axel

Release Management

Bug # or Issue	Status	ETA	Contact
Need to change the process of approving for Beta to first go through Aurora, for sake of risk mitigation
We need to watch for more stretch across multiple versions of beta than we currently have across weekly betas
Feature request: getting BuildIDs in Input to get more data on where feedback is coming from
What's new page - either always on or always off but we'll need a better plan

Metrics

Bug # or Issue	Status	ETA	Contact
No current issues	on track

Product Marketing & PR

Bug # or Issue	Status	ETA	Contact
Updating all channel comms (blog posts, etc)	on track		lforrest
Updating release notes to work for all audiences/communications	on track		lforrest
Naming of this change (DONE: 'daily betas', this is not-news, we aren't doing anything marketing-wise for this)	on track		lforrest

Legend

	Healthy: work is progressing as expected.
	Blocked: work is currently blocked.
	At Risk: work is at risk of missing the targeted completion date.
ETA	Estimated date for completion of the current task. Overall ETA for this work is 7/17.