Release Management/Process coordination for handling off-train releases

Goal

This document lists the high-level process (purpose, decisions, triggers, steps and people involved) for handling off-train release requests.

The goal of this document is to provide a single place where stakeholders (PM, EPM, etc) can have a quick idea of the various options available to ship changes to Firefox users.

The list of off-train release requests include (but are not limited to):

• Shipping content via Remote Settings
• Shipping a new Mozilla webextension
• GoFaster for shipping new system add-ons
• Blocklisting
• Shield studies
• Feature toggle (enable/disable) via Normandy
• Dot releases
• Chemspills

Many of these processes have similar workflow and teams of people involved.

For every process, the following roles will be involved in shipping the change:

✅ Development owner ✅ QA owner ✅ Release owner

Off-train release	Purpose	Trigger	Extra people involved	Checklist
Content sent via Remote Settings	Publishing content to Firefox users with the in-product messaging system (Remote Settings) Example: (TBD) Typical time-to-ship: (TBD)	Product manager/Dev owner files a bug and adds the messaging-system-request flag, filling out the resulting Bugzilla comment form. Driver: Product owner	✅ Product owner ✅ Publishing engineer ✅ Messaging (Firefox) Engineering peer ✅ Messaging Product owner ✅ Relman owner	Messaging system intake and release process
New Mozilla webext	A business need to publish Mozilla owned webext to Firefox Desktop users via AMO. Example: Facebook container Typical time-to-ship: 5 days	Product manager/Dev owner sends an email to engineering, Product Integrity leads (QA, security, release management) describing the business need and a rough sense of timeline to ship a new Webext. Driver: Product owner	✅ Product owner ✅ Data Steward owner ✅ Releng owner
GoFaster	Ship a new system add-on (SAO) or update an existing one Time-to-ship SAO: days to week depending on urgency Time-to-ship update: 5 days	An Intent to Implement email is sent to release-drivers by Product or Engineering Driver: Dev owner	✅ Product owner ✅ Data Steward owner ✅ Releng owner
Funnelcake builds	Funnelcake builds are special builds pushed to a small percentage of new release users with the aim of A/B testing via the downloads page. Funnelcake changes our download and install funnel for new users. Example: Onboarding Typical time-to-ship: Weeks	Funnelcake owner sends a funnelcake build “intent to ship” email to release-drivers. The email describes what’s been done so far and proposes a requested timeline to go live. Driver: Funnelcake owner	✅ Funnelcake owner ✅ Data Steward owner ✅ Releng owner
Blocklisting	In some specific cases, we can disable errant add-ons and other third-party software impact Firefox quality, privacy and security. Example: Stylish extension	A release blocking issue has been identified that needs to be addressed via blocklisting	✅ Blocklist master
Shield studies	Shield studies are controlled A/B tests. They are available on all channels, namely Nightly, Beta, Release. A/B experiments are done to understand user behavior, user retention, onboarding, test feature ideas, etc. Example: TLS 1.3 Time to ship new study: Weeks Time to ship updated study: 5-10 days	Shield team emails release-drivers mailing list with an “Intent to Ship” that describes shield study type, goal, target channel, rollout plan, QA sign off status and timeline. Driver: Product owner requesting the study	✅ Product owner ✅ Shield core team ✅ Firefox peer ✅ Data Steward
Feature toggle via Normandy	1) After go-live, some release blocking issues can be addressed via pref flips. 2) New feature can have a controlled rollout via pref flipping. Example: RDL, TLS 1.3 fallback	1) Product or Engineering or PI know of a release blocking 2) Product or PI team recommend controlled rollout of a feature post launch. Driver: Release owner	✅ Product owner
Dot releases	Dot release builds are pushed to release/ESR end-user users to mitigate release blocking issues Typical time-to-ship: 5 days	Product or Engineering or PI know of one or more release blocking issues and email/slack/IRC relman team for dot release planning and coordination. Driver: release owner	✅ Releng owner ✅ Sheriff ✅ Security owner (optional) ✅ Product owner (optional) ✅ Marketing owner (Optional)	Will be the same as chemspill
Chemspills	Chemspill is a special dot release, triggered by a security driven issue or active exploit in the wild Typical time-to-ship: 1 day or less	Product or Engineering or PI know of a release blocking security issue that needs to be addressed in a time-sensitive manner Driver: Chemspill owner	✅ Chemspill owner ✅ Releng owner ✅ Sheriff ✅ Product owner	Chemspill release checklist

General process

As all these processes are similar, the general process is described. Specific details are described in dedicated checklists.

1. Kick-off email thread or meeting to establish scope, owners and timeline.
2. Identify which off-train release vehicle to use. E.g. Controlled rollout or dot release or chemspill or shield study.
3. Create checklist, find owners and track items on the checklist.
4. Requirement doc - defines the goal of the change, why we are doing it and what will be done
5. Test plan - Using the requirement doc, the QA owner will draft the test plan. Test plan to be reviewed by product owner, eng owner.
6. Prepare the technical changes (code, pref change, etc)
7. “Build it” process (including signature if needed)
8. QA sign off - Formal QA sign off.
9. Rollout plan and timing - From the kick-off meeting to daily stand-ups, keep the team aligned on rollout plan and timing.
10. Communication - Do we need to communicate on the change? Release notes? Blog? If yes, identify owner and timeline to publish blog live.
11. Publication of the change
12. QA sign off of the publication of the change

Process: Messaging to users with Remote Settings

Extra information

Messaging system intake and release process

Process: Shipping a New Mozilla Webextension

A business need to publish Mozilla owned webext to Firefox Desktop users via AMO.

Decisions

• Products supported: Desktop / Fennec / both
• Platforms supported
• Firefox versions supported
• Release readiness criteria
  • QA
  • Performance
  • Security review
• What kind of telemetry data will this webext capture and send to Mozilla?
• Timeline and rollout strategy

Extra steps

1. Data Steward review and sign off - Review webext by a data steward to ensure we are not collecting any privacy data that we ought not to.
2. Security review - Security team within Product Integrity (PI) group will review and sign off on webext
3. Perf review - Perf team within Product Integrity (PI) group will review and sign off on webext
4. Maintain webext and fix issues reported by users - This happens in iterations until the time Mozilla plans to support this new webext

Checklist

Item	Owner	By When	Status	Completion Date
Kick-off meeting	<product-owner>
Webext rqmts doc	<dev-owner>
Test Plan	<qa-owner>
Coding	<dev-owner>
Testing	<qa-owner>
Security review	<security-owner>
Perf review	<perf-owner>
Data steward review	<data-steward>
Webext peer review	<webext-owner>
QA sign off	<qa-owner>
Go/NoGo decision	<product-owner>
Upload webext on AMO	<amo-owner>

Process: GoFaster release Ship a new system add-on (SAO) or update an existing one

Decisions

• Is GoFaster the appropriate release strategy?
• By nature, the rest of the decisions are the same as a Mozilla Webextension

Extra steps

1. By nature, the rest of the decisions are the same as a Mozilla Webextension

Checklists

Item	Owner	By When	Status	Completion Date
Kick-off meeting	<product-owner>
System addon rqmts doc	<dev-owner>
Test Plan	<qa-owner>
Coding	<dev-owner>
Testing	<qa-owner>
Security review	<security-owner>
Perf review	<perf-owner>
Data steward review (if relevant)	<data-steward>
Webext peer review	<webext-owner>
QA sign off	<qa-owner>
Go/NoGo decision	<product-owner>
Upload System addon on AMO	<amo-owner>

Extra information

The steps for shipping a system add-on are described in detail here:

https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process

Process: Funnelcake release

Funnelcake builds are special builds pushed to a small percentage of new release users with the aim of A/B testing via the downloads page. Funnelcake changes our download and install funnel for new users.

Decisions

• Timeline
• Team
• Scope
• New User population that will be targeted
• Release readiness criteria
  • QA
  • Perf
  • Security review
  • Data & Privacy review

Extra steps

1. Intent to ship email to release-drivers with the bug number
2. Releng owner to configure balrog rules and sign offs

Checklists

Item	Owner	By When	Status	Completion Date
Kick-off meeting	<product-owner>
Funnelcake rqmts doc	<dev-owner>
Test Plan	<qa-owner>
Coding	<dev-owner>
Testing	<qa-owner>
QA sign off	<qa-owner>
Go/NoGo decision	<product-owner>
Downloads page is ready and live	<amo-owner>

Extra information

https://wiki.mozilla.org/Funnelcake

Process: Add-ons/ThirdParty Blocklisting

In some specific cases, we can disable errant add-ons and other third-party software that negatively impact Firefox quality, privacy and security.

Decisions

Decide if we want to block an addons or a third party.

As this can have some important ripple effects, please be careful.

Extra steps

1. Intent to ship email to release-drivers to explain the change
2. The change should be published on the relevant platform

Checklists

Item	Owner	By When	Status	Completion Date
Request filed	<undefined>
Reach out to vendor/developer	<product-owner>
Meeting to confirm the blocking	<release-owner>
Development	<dev-owner>
Testing	<qa-owner>
QA sign off	<qa-owner>
Publication of the change	<amo-owner>

Extra information

https://wiki.mozilla.org/Blocklisting

Process: Shield studies

Shield studies are controlled A/B tests, using either a pref-flip or an add-on. They are available on all channels, namely Nightly, Beta, Release. A/B experiments are done to understand user behavior, user retention, onboarding, test feature ideas, etc.

Decisions

• Work with Data Science first to determine the best way to answer your questions. A pref-flip or add-on study is only one way to find answers. Guidance to get started with Data Science is here.
• If Data Science and product determine a pref-flip or add-ons study is needed, you will fill out Experimenter together to get started. Getting Started guidance here.
• These are the stages between starting Experimenter and the study releasing.

Extra steps

1. In Experimenter, after you have completed your "Draft", click the "Ready for Sign-offs" button to get the checklist of Required and Optional items needed to launch your study.
2. When you have all your required and optional checklist items complete, click the "Ready to Ship" button to let Normandy know this is ready to be launched. You will be contacted with any questions.

Checklists

Item	Owner	By When	Status	Completion Date
Data Science Bug	product-owner
Analysis Battery	data scientist
Experimenter Draft	product-owner & data scientist
Complete Study checklist	product-owner
Kick-off the study	Normandy-owner
Study Results	product-owner & data scientist

Extra information

Useful links

• risk assessment criteria
• Experimenter Guidance

Process: Feature rollout via Normandy

1) After go-live, some release blocking issues can be addressed via pref flips.

2) New feature can have a controlled rollout via pref flipping.

Decisions

• Timeline (Refer to the feature rollout playbook for help with this)
• Products affected: Desktop, Fennec, ESR?
• Platforms affected: All, Windows, Mac, Linux?
• Rollout strategy

Extra steps

1. File a bug for handling the rollout (examples: bug 1467514 or bug 1523978)
2. Development owner (likely mythmon) configures the recipe in Normandy to flip pref for targeted end-users
3. For release-unblockers/hotfixes, set up on stage, so QA can test on https://delivery-console.stage.mozaws.net
4. Release owner reviews, approves and publishes recipe
5. Feature team, release owner monitor data to verify:
   1. Recipe uptake is going as planned
   2. Feature is having the intended effect on end-users

Checklist

<TBD>

Process Dot Release

Dot release builds are pushed to release and/or ESR end-user users to mitigate release blocking issues.

Decisions

• Timeline
• Bugs that are dot release drivers and ride-alongs
• Products affected: Desktop, Fennec, ESR?
• Platforms affected: All, Windows, Mac, Linux?
• Rollout strategy
• Release notes, sec advisories, comms plan, What’s New Page

Extra steps

1. Release owner drafts release notes and shares for review.
2. Security owner drafts security advisory as needed and shares for review.

Checklist

Example: 59.0.x dot release checklist

Process: Chemspills

Chemspill is a special dot release, triggered by a security driven issue or active exploit in the wild

Decisions

• Timeline
• Products affected: Desktop, Fennec, ESR?
• Platforms affected: All, Windows, Mac, Linux?
• sec advisories, comms plan

Extra steps

1. Security owner drafts security advisory as needed and shares for review.
2. Create a CVE if necessary
3. Reach out to partners and/or other projects impacted if needed

Checklist

Chemspill release checklist

Extra information

Chemspill process description