Release Management/Process coordination for handling off-train releases
Goal
This document lists the high-level process (purpose, decisions, triggers, steps and people involved) for handling off-train release requests.
The goal of this document is to provide a single place where stakeholders (PM, EPM, etc) can have a quick idea of the various options available to ship changes to Firefox users.
The list of off-train release requests include (but are not limited to):
- Shipping a new Mozilla webextension
- GoFaster for shipping new system add-ons
- Blocklisting
- Shield studies
- Feature toggle (enable/disable) via Normandy
- Dot releases
- Chemspills
Many of these processes have similar workflow and teams of people involved.
For every process, the following roles will be involved in shipping the change:
✅ Development owner ✅ QA owner ✅ Release owner
| Off-train release | Purpose | Trigger | Extra people involved | Checklist |
| New Mozilla webext | A business need to publish Mozilla owned webext to Firefox Desktop users via AMO.
Example: Facebook container
|
Product manager/Dev owner sends an email to engineering, Product Integrity leads (QA, security, release management) describing the business need and a rough sense of timeline to ship a new Webext.
|
✅ Product owner
✅ Data Steward owner ✅ Releng owner
|
TODO add a link to a template for the process |
| GoFaster | Ship a new system add-on (SAO) or update an existing one
Time-to-ship update: 5 days |
An Intent to Implement email is sent to release-drivers by Product or Engineering
|
✅ Product owner
✅ Data Steward owner ✅ Releng owner
|
|
| Funnelcake builds | Funnelcake builds are special builds pushed to a small percentage of new release users with the aim of A/B testing via the downloads page. Funnelcake changes our download and install funnel for new users.
|
Funnelcake owner sends a funnelcake build “intent to ship” email to release-drivers. The email describes what’s been done so far and proposes a requested timeline to go live.
|
✅ Funnelcake owner
✅ Data Steward owner ✅ Releng owner
|
|
| Blocklisting | In some specific cases, we can disable errant add-ons and other third-party software impact Firefox quality, privacy and security.
|
A release blocking issue has been identified that needs to be addressed via blocklisting | ✅ Blocklist master | |
| Shield studies | Shield studies are controlled A/B tests. They are available on all channels, namely Nightly, Beta, Release. A/B experiments are done to understand user behavior, user retention, onboarding, test feature ideas, etc.
|
Shield team emails release-drivers mailing list with an “Intent to Ship” that describes shield study type, goal, target channel, rollout plan, QA sign off status and timeline.
|
✅ Product owner
✅ Shield core team ✅ Firefox peer ✅ Data Steward
|
|
| Feature toggle via Normandy | 1) After go-live, some release blocking issues can be addressed via pref flips.
2) New feature can have a controlled rollout via pref flipping.
|
1) Product or Engineering or PI know of a release blocking
|
✅ Product owner | |
| Dot releases | Dot release builds are pushed to release/ESR end-user users to mitigate release blocking issues
|
Product or Engineering or PI know of one or more release blocking issues and email/slack/IRC relman team for dot release planning and coordination.
|
✅ Releng owner
✅ Sheriff ✅ Security owner (optional) ✅ Product owner (optional) ✅ Marketing owner (Optional)
|
Will be the same as chemspill |
| Chemspills | Chemspill is a special dot release, triggered by a security driven issue or active exploit in the wild
|
Product or Engineering or PI know of a release blocking security issue that needs to be addressed in a time-sensitive manner
|
✅ Chemspill owner
✅ Releng owner ✅ Sheriff ✅ Product owner |
Chemspill release checklist |
2 General process
As all these processes are similar, the general process is describe. Specificities are described in dedicated checklists.
- Kick-off email thread or meeting to establish scope, owners and timeline.
- Identify which off-train release vehicle to use. E.g. Controlled rollout or dot release or chemspill or shield study.
- Create checklist, find owners and track items on the checklist.
- Requirement doc - defines the goal of the change, why we are doing it and what will be done
- Test plan - Using the requirement doc, the QA owner will draft the test plan. Test plan to be reviewed by product owner, eng owner.
- Prepare the technical changes (code, pref change, etc)
- “Build it” process (including signature if needed)
- QA sign off - Formal QA sign off.
- Rollout plan and timing - From the kick-off meeting to daily stand-ups, keep the team aligned on rollout plan and timing.
- Communication - Do we need to communicate on the change? Release notes? Blog? If yes, identify owner and timeline to publish blog live.
- Publication of the change
- QA sign off of the publication of the change
Process: Shipping a New Mozilla Webextension
A business need to publish Mozilla owned webext to Firefox Desktop users via AMO.
Decisions
- Products supported: Desktop / Fennec / both
- Platforms supported
- Firefox versions supported
- Release readiness criteria
- QA
- Performance
- Security review
- What kind of telemetry data will this webext capture and send to Mozilla?
- Timeline and rollout strategy
Extra steps
- Data Steward review and sign off - Review webext by a data steward to ensure we are not collecting any privacy data that we ought not to.
- Security review - Security team within Product Integrity (PI) group will review and sign off on webext
- Perf review - Perf team within Product Integrity (PI) group will review and sign off on webext
- Maintain webext and fix issues reported by users - This happens in iterations until the time Mozilla plans to support this new webext
Checklist
| Item | Owner | By When | Status | Completion Date |
| Kick-off meeting | <product-owner> | |||
| Webext rqmts doc | <dev-owner> | |||
| Test Plan | <qa-owner> | |||
| Coding | <dev-owner> | |||
| Testing | <qa-owner> | |||
| Security review | <security-owner> | |||
| Perf review | <perf-owner> | |||
| Data steward review | <data-steward> | |||
| Webext peer review | <webext-owner> | |||
| QA sign off | <qa-owner> | |||
| Go/NoGo decision | <product-owner> | |||
| Upload webext on AMO | <amo-owner> |
Process: GoFaster release Ship a new system add-on (SAO) or update an existing one
Decisions
- Is GoFaster the appropriate release strategy?
- By nature, the rest of the decisions are the same as a Mozilla Webextension
Extra steps
By nature, the rest of the decisions are the same as a Mozilla Webextension
Checklists
| Item | Owner | By When | Status | Completion Date |
| Kick-off meeting | <product-owner> | |||
| System addon rqmts doc | <dev-owner> | |||
| Test Plan | <qa-owner> | |||
| Coding | <dev-owner> | |||
| Testing | <qa-owner> | |||
| Security review | <security-owner> | |||
| Perf review | <perf-owner> | |||
| Data steward review (if relevant) | <data-steward> | |||
| Webext peer review | <webext-owner> | |||
| QA sign off | <qa-owner> | |||
| Go/NoGo decision | <product-owner> | |||
| Upload System addon on AMO | <amo-owner> |
Extra information
https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process
Process: Funnelcake release
Funnelcake builds are special builds pushed to a small percentage of new release users with the aim of A/B testing via the downloads page. Funnelcake changes our download and install funnel for new users.
Decisions
- Timeline
- Team
- Scope
- New User population that will be targeted
- Release readiness criteria
- QA
- Perf
- Security review
- Data & Privacy review
Extra steps
- Intent to ship email to release-drivers with the bug number
- Releng owner to configure balrog rules and sign offs
Checklists
| Item | Owner | By When | Status | Completion Date |
| Kick-off meeting | <product-owner> | |||
| Funnelcake rqmts doc | <dev-owner> | |||
| Test Plan | <qa-owner> | |||
| Coding | <dev-owner> | |||
| Testing | <qa-owner> | |||
| QA sign off | <qa-owner> | |||
| Go/NoGo decision | <product-owner> | |||
| Downloads page is ready and live | <amo-owner> |
Extra information
https://wiki.mozilla.org/Funnelcake
Process: Add-ons/ThirdParty Blocklisting
In some specific cases, we can disable errant add-ons and other third-party software that negatively impact Firefox quality, privacy and security.
Decisions
Decide if we want to block an addons or a third party.
As this can have some important ripple effects, please be careful.
Extra steps
- Intent to ship email to release-drivers to explain the change
- The change should be published on the relevant platform
Checklists
| Item | Owner | By When | Status | Completion Date |
| Request filed | <undefined> | |||
| Reach out to vendor/developer | <product-owner> | |||
| Meeting to confirm the blocking | <release-owner> | |||
| Development | <dev-owner> | |||
| Testing | <qa-owner> | |||
| QA sign off | <qa-owner> | |||
| Publication of the change | <amo-owner> |
Extra information
https://wiki.mozilla.org/Blocklisting
Process: Shield studies
Shield studies are controlled A/B tests. They are available on all channels, namely Nightly, Beta, Release. A/B experiments are done to understand user behavior, user retention, onboarding, test feature ideas, etc.
Decisions
- Target channel
- Rollout plan (channel, user population, test/control channel breakdown)
- Timeline
- Data/Legal review
Extra steps
- Product Hypothesis Doc (Phd) review
Checklists
| Item | Owner | By When | Status | Completion Date |
| Phd review | <product-owner> | |||
| Data steward/Privacy review | <data-steward> | |||
| Firefox peer review | <shield-owner> | |||
| QA sign off | <qa-owner> | |||
| Relman Go/NoGo sign off | <relman> | |||
| Kick-off the study | <shield-owner> |
Extra information
Useful links
Process: Feature rollout via Normandy
1) After go-live, some release blocking issues can be addressed via pref flips.
2) New feature can have a controlled rollout via pref flipping.
Decisions
- Timeline
- Products affected: Desktop, Fennec, ESR?
- Platforms affected: All, Windows, Mac, Linux?
- Rollout strategy
Extra steps
- Development owner configures the recipe in Normandy to flip pref for targeted end-users
- Release owner reviews, approves and publishes recipe
- Feature team, release owner monitor data to verify:
- Recipe uptake is going as planned
- Feature is having the intended effect on end-users
Checklist
<TBD>
Process Dot Release
Dot release builds are pushed to release and/or ESR end-user users to mitigate release blocking issues.
Decisions
- Timeline
- Bugs that are dot release drivers and ride-alongs
- Products affected: Desktop, Fennec, ESR?
- Platforms affected: All, Windows, Mac, Linux?
- Rollout strategy
- Release notes, sec advisories, comms plan, What’s New Page
Extra steps
- Release owner drafts release notes and shares for review.
- Security owner drafts security advisory as needed and shares for review.
Checklist
Example: 59.0.x dot release checklist
Process: Chemspills
Chemspill is a special dot release, triggered by a security driven issue or active exploit in the wild
Decisions
- Timeline
- Products affected: Desktop, Fennec, ESR?
- Platforms affected: All, Windows, Mac, Linux?
- sec advisories, comms plan
Extra steps
- Security owner drafts security advisory as needed and shares for review.
- Create a CVE if necessary
- Reach out to partners and/or other projects impacted if needed