Release Management/Process coordination for handling off-train releases

Goal

This document lists the high-level process (purpose, decisions, triggers, steps and people involved) for handling off-train release requests.

The goal of this document is to provide a single place where stakeholders (PM, EPM, etc) can have a quick idea of the various options available to ship changes to Firefox users.

The list of off-train release requests include (but are not limited to):

• Shipping a new Mozilla webextension
• GoFaster for shipping new system add-ons
• Blocklisting
• Shield studies
• Feature toggle (enable/disable) via Normandy
• Dot releases
• Chemspills

Many of these processes have similar workflow and teams of people involved.

For every process, the following roles will be involved in shipping the change:

✅ Development owner ✅ QA owner ✅ Release owner

Off-train release	Purpose	Trigger	Extra people involved	Checklist
New Mozilla webext	A business need to publish Mozilla owned webext to Firefox Desktop users via AMO. Example: Facebook container Typical time-to-ship: 5 days	Product manager/Dev owner sends an email to engineering, Product Integrity leads (QA, security, release management) describing the business need and a rough sense of timeline to ship a new Webext. Driver: Product owner	✅ Product owner ✅ Data Steward owner ✅ Releng owner	TODO add a link to a template for the process
GoFaster	Ship a new system add-on (SAO) or update an existing one Time-to-ship SAO: days to week depending on urgency Time-to-ship update: 5 days	An Intent to Implement email is sent to release-drivers by Product or Engineering Driver: Dev owner	✅ Product owner ✅ Data Steward owner ✅ Releng owner
Funnelcake builds	Funnelcake builds are special builds pushed to a small percentage of new release users with the aim of A/B testing via the downloads page. Funnelcake changes our download and install funnel for new users. Example: Onboarding Typical time-to-ship: Weeks	Funnelcake owner sends a funnelcake build “intent to ship” email to release-drivers. The email describes what’s been done so far and proposes a requested timeline to go live. Driver: Funnelcake owner	✅ Funnelcake owner ✅ Data Steward owner ✅ Releng owner
Blocklisting	In some specific cases, we can disable errant add-ons and other third-party software impact Firefox quality, privacy and security. Example: Stylish extension	A release blocking issue has been identified that needs to be addressed via blocklisting	✅ Blocklist master
Shield studies	Shield studies are controlled A/B tests. They are available on all channels, namely Nightly, Beta, Release. A/B experiments are done to understand user behavior, user retention, onboarding, test feature ideas, etc. Example: TLS 1.3 Time to ship new study: Weeks Time to ship updated study: 5-10 days	Shield team emails release-drivers mailing list with an “Intent to Ship” that describes shield study type, goal, target channel, rollout plan, QA sign off status and timeline. Driver: Product owner requesting the study	✅ Product owner ✅ Shield core team ✅ Firefox peer ✅ Data Steward
Feature toggle via Normandy	1) After go-live, some release blocking issues can be addressed via pref flips. 2) New feature can have a controlled rollout via pref flipping. Example: RDL, TLS 1.3 fallback	1) Product or Engineering or PI know of a release blocking 2) Product or PI team recommend controlled rollout of a feature post launch. Driver: Release owner	✅ Product owner
Dot releases	Dot release builds are pushed to release/ESR end-user users to mitigate release blocking issues Typical time-to-ship: 5 days	Product or Engineering or PI know of one or more release blocking issues and email/slack/IRC relman team for dot release planning and coordination. Driver: release owner	✅ Releng owner ✅ Sheriff ✅ Security owner (optional) ✅ Product owner (optional) ✅ Marketing owner (Optional)	Will be the same as chemspill
Chemspills	Chemspill is a special dot release, triggered by a security driven issue or active exploit in the wild Typical time-to-ship: 1 day or less	Product or Engineering or PI know of a release blocking security issue that needs to be addressed in a time-sensitive manner Driver: Chemspill owner	✅ Chemspill owner ✅ Releng owner ✅ Sheriff ✅ Product owner	Chemspill release checklist

General process

As all these processes are similar, the general process is describe. Specificities are described in dedicated checklists.

1. Kick-off email thread or meeting to establish scope, owners and timeline.
2. Identify which off-train release vehicle to use. E.g. Controlled rollout or dot release or chemspill or shield study.
3. Create checklist, find owners and track items on the checklist.
4. Requirement doc - defines the goal of the change, why we are doing it and what will be done
5. Test plan - Using the requirement doc, the QA owner will draft the test plan. Test plan to be reviewed by product owner, eng owner.
6. Prepare the technical changes (code, pref change, etc)
7. “Build it” process (including signature if needed)
8. QA sign off - Formal QA sign off.
9. Rollout plan and timing - From the kick-off meeting to daily stand-ups, keep the team aligned on rollout plan and timing.
10. Communication - Do we need to communicate on the change? Release notes? Blog? If yes, identify owner and timeline to publish blog live.
11. Publication of the change
12. QA sign off of the publication of the change

Process: Shipping a New Mozilla Webextension

A business need to publish Mozilla owned webext to Firefox Desktop users via AMO.

Decisions

Products supported: Desktop / Fennec / both

Platforms supported

Firefox versions supported

Release readiness criteria

• QA
• Performance
• Security review

What kind of telemetry data will this webext capture and send to Mozilla?

Timeline and rollout strategy

3.2 Extra steps

1. Data Steward review and sign off - Review webext by a data steward to ensure we are not collecting any privacy data that we ought not to.
2. Security review - Security team within Product Integrity (PI) group will review and sign off on webext
3. Perf review - Perf team within Product Integrity (PI) group will review and sign off on webext
4. Maintain webext and fix issues reported by users - This happens in iterations until the time Mozilla plans to support this new webext

Item	Owner	By When	Status	Completion Date
Kick-off meeting	<product-owner>
Webext rqmts doc	<dev-owner>
Test Plan	<qa-owner>
Coding	<dev-owner>
Testing	<qa-owner>
Security review	<security-owner>
Perf review	<perf-owner>
Data steward review	<data-steward>
Webext peer review	<webext-owner>
QA sign off	<qa-owner>
Go/NoGo decision	<product-owner>
Upload webext on AMO	<amo-owner>

Process: GoFaster release Ship a new system add-on (SAO) or update an existing one

Decisions

Is GoFaster the appropriate release strategy?

By nature, the rest of the decisions are the same as a Mozilla Webextension

Extra steps

By nature, the rest of the decisions are the same as a Mozilla Webextension

4As I go through a final review of this, I am leaning towards removing the checklists section. I think as a first-pass of documenting the various process, we should let the teams decide what kind of checklist they want to maintain, in what form and use which repository. If the feedback we get is that checklists would make this more efficient, we can come back to it in a phase2. Thoughts? +yor@mozilla.comMove that into a spreadsheet.3 Checklists

Item	Owner	By When	Status	Completion Date
Kick-off meeting	<product-owner>
System addon rqmts doc	<dev-owner>
Test Plan	<qa-owner>
Coding	<dev-owner>
Testing	<qa-owner>
Security review	<security-owner>
Perf review	<perf-owner>
Data steward review (if relevant)	<data-steward>
Webext peer review	<webext-owner>
QA sign off	<qa-owner>
Go/NoGo decision	<product-owner>
Upload System addon on AMO	<amo-owner>

Extra information

https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process

Process: Funnelcake release

Funnelcake builds are special builds pushed to a small percentage of new release users with the aim of A/B testing via the downloads page. Funnelcake changes our download and install funnel for new users.

Decisions

• Timeline
• Team
• Scope
• New User population that will be targeted
• Release readiness criteria
  • QA
  • Perf
  • Security review
  • Data & Privacy review

Extra steps

Intent to ship email to release-drivers with the bug number

Releng owner to configure balrog rules and sign offs

Checklists

Item	Owner	By When	Status	Completion Date
Kick-off meeting	<product-owner>
Funnelcake rqmts doc	<dev-owner>
Test Plan	<qa-owner>
Coding	<dev-owner>
Testing	<qa-owner>
QA sign off	<qa-owner>
Go/NoGo decision	<product-owner>
Downloads page is ready and live	<amo-owner>

Extra information

https://wiki.mozilla.org/Funnelcake

Process: Add-ons/ThirdParty Blocklisting

In some specific cases, we can disable errant add-ons and other third-party software that negatively impact Firefox quality, privacy and security.

Decisions

Decide if we want to block an addons or a third party.

As this can have some important ripple effects, please be careful.

Extra steps

1. Intent to ship email to release-drivers to explain the change
2. The change should be published on the relevant platform

Checklists

Item	Owner	By When	Status	Completion Date
Request filed	<undefined>
Reach out to vendor/developer	<product-owner>
Meeting to confirm the blocking	<release-owner>
Development	<dev-owner>
Testing	<qa-owner>
QA sign off	<qa-owner>
Publication of the change	<amo-owner>

Extra information

https://wiki.mozilla.org/Blocklisting

Process: Shield studies

Shield studies are controlled A/B tests. They are available on all channels, namely Nightly, Beta, Release. A/B experiments are done to understand user behavior, user retention, onboarding, test feature ideas, etc.

Decisions

• Target channel
• Rollout plan (channel, user population, test/control channel breakdown)
• Timeline
• Data/Legal review

Extra steps

1. Product Hypothesis Doc (Phd) review

Checklists

Item	Owner	By When	Status	Completion Date
Phd review	<product-owner>
Data steward/Privacy review	<data-steward>
Firefox peer review	<shield-owner>
QA sign off	<qa-owner>
Relman Go/NoGo sign off	<relman>
Kick-off the study	<shield-owner>

Extra information

Useful links*

Release management risk assessment criteria

• https://wiki.mozilla.org/Firefox/Shield/Shield_Studies

Process: Feature rollout via Normandy

1) After go-live, some release blocking issues can be addressed via pref flips.

2) New feature can have a controlled rollout via pref flipping.

Decisions

• Timeline
• Products affected: Desktop, Fennec, ESR?
• Platforms affected: All, Windows, Mac, Linux?
• Rollout strategy

Extra steps

• Development owner configures the recipe in Normandy to flip pref for targeted end-users
• Release owner reviews, approves and publishes recipe
• Feature team, release owner monitor data to verify:
  • Recipe uptake is going as planned
  • Feature is having the intended effect on end-users

Checklist

<TBD>

Process Dot Release

Dot release builds are pushed to release/ESR end-user users to mitigate release blocking issues

Decisions

• Timeline
• Bugs that are dot release drivers and ride-alongs
• Products affected: Desktop, Fennec, ESR?
• Platforms affected: All, Windows, Mac, Linux?
• Rollout strategy
• Release notes, sec advisories, comms plan, What’s New Page

Extra steps

1. Release owner drafts release notes and shares for review.
2. Security owner drafts security advisory as needed and shares for review.

Checklist

Example: 59.0.x dot release checklist

Process: Chemspills

Chemspill is a special dot release, triggered by a security driven issue or active exploit in the wild

Decisions

• Timeline
• Products affected: Desktop, Fennec, ESR?
• Platforms affected: All, Windows, Mac, Linux?
• sec advisories, comms plan

Extra steps

1. Security owner drafts security advisory as needed and shares for review.
2. Create a CVE if necessary
3. Reach out to partners and/or other projects impacted if needed

checklist

Chemspill release checklist

Extra information

• Chemspill process description