Compatibility/System Addon/Release Process

Interventions Release Process

Note: for simplicity, interventions and UA overrides will be referred to solely as interventions in this section.

Intervention Types

There are two types of interventions that we ship in Firefox: high priority and low priority. As a general rule of thumb, a low priority intervention will fix a long-standing bug. A high priority intervention will fix a regression in a top site. In case of confusion, please get in touch with Mike Taylor for clarification as to which class we’re dealing with.

Low priority interventions should be landed in the Mozilla webcompat-addon repo on GitHub and upstreamed to the relevant tree once per release. These do not require outside collaboration or approval to ship (except for the case of Fennec ESR, which will require Relman uplift to the ESR branch while it exists as a supported product), beyond WebCompat Addon module peer or owner approval.

High priority interventions require shipping out-of-band via Balrog (or Normandy in the future), and as such, require a more in-depth QA and Relman release process, at least for Fennec and Desktop. The process for Release Fenix is currently undefined.

Shipping Scenarios

All interventions should first land in the Mozilla webcompat-addon repo on GitHub and be accompanied by a version number bump in the addon’s manifest.json. There may be high priority situations where it’s more desirable to land in Mozilla Central first. In these cases, the patches must be backported to GitHub as soon as the intervention is shipped.

Once the intervention is reviewed by a [WebCompat Addon module peer] and landed, one or more of the following shipping scenarios is followed, depending on the affected platforms and products:

Desktop Releases

Low Priority

1. Make sure the version number in manifest.json is incremented.
2. Land the updated addon version in Mozilla Central via a bug in Web Compatibility::Interventions
   1. Export the sources into mozilla-central using the export-mc jake command.
   2. Create a patch for mozilla-central, upload it to Phabricator, and have a WebCompat Addon module peer review that.
3. That’s it. Let the code ride the trains.

High Priority

[TODO] make sure this is up to date with the new mozilla-extensions github actions workflow

1. File a bug to ship the addon via Balrog in the Web Compatibility::Interventions component
2. Attach an XPI for the WebCompat QA team to test locally using the following naming convention: webcompat-N.N.N.unsigned.xpi
3. When testing, QA can use an unbranded build, or Nightly.
4. Describe how QA can test and verify the intervention in a comment
5. QA will test and leave a comment with their findings: either sign-off, or describing unexpected behavior that needs to be investigated.
6. Request the signed addon be staged to the release-sysaddon channel by :rdalal
7. Ask QA to verify the release-sysaddon update works (instructions here), and the intervention works as expected
8. Once QA signs off, request Relman for signoff in Balrog (a needinfo request in the bug is sufficient)
9. Once the addon ships to release:
   1. Land the addon in Mozilla Central and request Beta uplift
   2. File a bug to remove the addon from Balrog in the Firefox::System Add-ons: Off-train Deployment component. This should happen when the next addon version rides to release, otherwise it will be overridden by the older version in Balrog. e.g., Bug 1587186.

Mobile Releases

Fennec ESR

The process for Fennec will match Desktop, except the patches need to be written against the ESR branch.

Low Priority

1. Relman will need to approve ESR uplift for the next ESR release once the patch has landed in the ESR branch. This happens in Bugzilla by requesting ESR approval for the patch.

High Priority

1. Follow the same steps as Desktop High Priority.

Fenix / Android Components

The system addon is served as a component. Currently Fenix pulls in the latest version, which means the updated addon should be served in the next Google Play Store update.

1. Export the sources into the android-components repo using the export-ac export command.
2. File an issue and open a pull request against the Android Components repo on GitHub which contains the updated addon and request review. Currently Dennis Schubert and Mike Taylor can review and merge, as well as any core member of the mozilla-mobile organization.
3. That’s it. The new version will be used when the new android-components version is released (currently weekly).

[TODO] depending on how updated android components are released in Fenix, there may not be a difference between high and low priority releases. But if there is, we need to find out how to ask for an urgent version release...ideally before we need it

Final Checklist when shipping a release

• [ ] Verify addon version has been incremented
• [ ] Verify new intervention(s) appears in about:compat as expected
• [ ] Close all Bugzilla bugs with interventions associated with the current release
• [ ] WebCompat QA has signed off on XPI
• [ ] Balrog folks have uploaded XPI
• [ ] Relman has signed off on XPI in Balrog
• [ ] Fennec ESR uplift request, when relevant
• [ ] Desktop ESR uplift request, when relevant
• [ ] m-c beta or ESR uplift request, when relevant
• [ ] Intent to Ship email is sent to release-drivers@ mailing list for high priority releases
• [ ] WebCompat QA marks each Bugzilla bug associated with the release as VERIFIED

Checklist for follow-ups after landing/deploying everywhere

When the new addon version is released everywhere and there will be no more changes to that version, there are a couple of follow-up steps to do in preparation for the next release cycle.

• [ ] Tag the version on GitHub with `git tag -a v8.0.0 -m 'v8.0.0'` and push the tags with `git push --tags` so we have a persistent record of "what shipped when" on GitHub as well.

Interventions Release Rotations

Engineers on the Web Compatibility team will rotate on ownership of shipping new versions of our interventions addons, serving as an Intervention Release Owner (IRO). The process will follow a predictable 4 week schedule, mirroring the proposed 4 week Firefox release schedule, however, we will attempt to land the interventions 1 week before soft freeze (see the schedule above for dates). Another way to look at this is 2 weeks before release.

A tracking bug for the next release should be filed in Bugzilla (in the Web Compatibility::Interventions component). Bugs for adding or removing interventions in the current release cycle should block this bug.

During bug diagnosis, if a site is identified as a low priority intervention candidate, a label is added for the next IRO to take care of during their rotation. Low priority interventions ride the trains without any need for uplifts or out of band shipping mechanisms. The expectation is that there will be a single regular low-priority release for each version of Firefox, driven by the IRO.

High priority interventions should be flagged immediately to the IRO who will then begin the process necessary to ship an off-train intervention.

Interventions Release Rotations

Engineers on the Web Compatibility team will rotate on ownership of shipping new versions of our interventions addons, serving as an Intervention Release Owner (IRO). The process will follow a predictable 4 week schedule, mirroring the proposed 4 week Firefox release schedule.

During bug diagnosis, if a site is identified as a low priority intervention candidate, a label shall be added for the IRO to take care of during their rotation. Low priority interventions ride the trains without any need for uplifts or out of band shipping mechanisms. The expectation is that there will be a single regular low-priority release for each version of Firefox, driven by the IRO.

High priority interventions should be flagged immediately to the IRO who will then begin the process necessary to ship an off-train intervention.

Candidates for interventions

There is a list of sites as well as a Bugzilla bug query for sites that need interventions. In addition, the `type-uaoverride` label may be useful to look at. During a rotation, you should look at both of these sites and determine which are the most important to work on and ship (or close, if appropriate).

Lists of currently deployed interventions across the products and channels are available here https://arewehotfixingthewebyet.com/.

In addition, there is also a `sitepatch-applied` label that helps to keep track of existing interventions. Take care to add this to bugs after shipping a site patch.

IRO Rotation Responsibilities

• Authoring, testing, landing high priority intervention patches
• Authoring, testing, landing normal priority interventions
• Requesting testing of high risk interventions from the WebCompat QA team
• Coordinating QA verification and release stakeholders (Balrog, AC, etc) for high priority interventions
• Sending an intent-to-ship email for high priority intervention patches
• Landing patches in GitHub, Mozilla Central, and Android Components repos
• Requesting uplifts for interventions when necessary
• Backporting interventions from mozilla-central where necessary (see Backporting a Mozilla Central Patch).
• Handling potential regression fallout from interventions
• Add sitepatch-applied labels to web-bugs issues, as relevant.

Intervention Release Owner Schedule

For simplicity, we rotate on a per-release following the Nightly schedule at [Release Calendar].

Upcoming Releases

• Firefox 87 (v19) - Tom 2021-01-21 2021Q1
• Firefox 88 (v20) - Dennis 2021-02-18 2021Q1
• Firefox 89 (v21) - Karl 2021-03-18 2021Q1
• Firefox 90 (v22) - Ksenia 2021-04-15 2021Q2
• Firefox 91 (v23) - Tom 2021-05-13 2021Q2
• Firefox 92 (v24) - Dennis 2021-06-10 2021Q2
• Firefox 93 (v25) - Karl 2021-07-08 2021Q3
• Firefox 94 (v26) - Ksenia 2021-08-05 2021Q3
• Firefox 95 (v26) - Tom 2021-09-02 2021Q3

Past Releases

• Firefox 75 (v7) - Ksenia
• Firefox 76 (v8) - Dennis
• Firefox 77 (v9) - Mike
• Firefox 78 (v10) - Ksenia
• Firefox 79 (v11) - Mike
• Firefox 80 (v12) - Dennis
• Firefox 81 (v13) - Karl
• Firefox 82 (v14) - Ksenia
• Firefox 83 (v15) - Tom
• Firefox 84 (v16) - Dennis
• Firefox 85 (v17) - Karl
• Firefox 86 (v18) - Ksenia