Compatibility/System Addon/Release Process

Interventions Releases

Module Owner: Dennis Schubert

Currently Released Interventions

A usually up-to-date list of interventions currently shipped can be found with a search for web-bugs issues labeled with "sitepatch-applied", and on Bugzilla with a search for issues with the "[webcompat:sitepatch-applied]" whiteboard tag.

Interventions Release Process

Note: for simplicity, interventions and UA overrides will be referred to solely as interventions in this section.

Intervention Types

There are two types of interventions that we ship in Firefox: high priority and low priority. As a general rule of thumb, a low priority intervention will fix a long-standing bug. A high priority intervention will fix a regression in a top site. In case of confusion, please get in touch with Dennis Schubert for clarification as to which class we’re dealing with.

Low priority interventions should be landed in the Mozilla webcompat-addon repo on GitHub and upstreamed to the relevant tree once per release. These do not require outside collaboration or approval to ship, beyond WebCompat Addon module peer or owner approval.

High priority interventions require shipping out-of-band via Balrog (or Normandy in the future), and as such, require a more in-depth QA and Relman release process.

Shipping Scenarios

All interventions should first land in the Mozilla webcompat-addon repo on GitHub and be accompanied by a version number bump in the addon’s manifest.json. There may be high priority situations where it’s more desirable to land in Mozilla Central first. In these cases, the patches must be backported to GitHub as soon as the intervention is shipped.

Once the intervention is reviewed by a WebCompat Addon module peer and landed, one or more of the following shipping scenarios is followed, depending on the affected platforms and products:

Desktop Releases

Low Priority

1. Make sure the version number in manifest.json is incremented.
2. Land the updated addon version in Mozilla Central via a bug in Web Compatibility::Interventions
   1. Export the sources into mozilla-central using the export-mc jake command.
   2. Create a patch for mozilla-central, upload it to Phabricator, and have a WebCompat Addon module peer review that.
3. That’s it. Let the code ride the trains.

High Priority

[TODO] make sure this is up to date with the new mozilla-extensions github actions workflow when work is done

Changes to the Interventions Add-on can be directly deployed to release-users, which allows for interventions that do not require a dot-release. System Addon rollouts go into effect immediately after launching it, with the first Firefox users picking up the update within an hour. The majority of users will have received the update within five days.

A current overview of the high-level steps required to ship via ShipIt can be found this Google Doc (Mozilla-internal). Ping :denschub if something needs to be rolled out urgently.

Fenix / Android Components

The system addon is served as a component. Currently Fenix pulls in the latest version, which means the updated addon should be served in the next Google Play Store update.

1. Export the sources into the android-components repo using the export-ac export command.
2. File an issue and open a pull request against the Android Components repo on GitHub which contains the updated addon and request review. Any core member of the mozilla-mobile organization is happy to help, but :csadilek frequently reviews our changes, so he should be the first point of contact if possible.
3. That’s it. The new version will be used when the new android-components version is released (currently weekly).

Firefox iOS

Firefox iOS has currently a very light UA override mechanism (at the time of this writing).

We need to define in cooperation with the Firefox iOS Team on how to remove the pain for users. Most of the issues are identified through webcompat reporting.

Final Checklist when shipping a release

• [ ] Verify addon version has been incremented
• [ ] Verify new intervention(s) appears in about:compat as expected
• [ ] Close all Bugzilla bugs with interventions associated with the current release
• [ ] WebCompat QA has signed off on XPI
• [ ] Balrog folks have uploaded XPI
• [ ] Relman has signed off on XPI in Balrog
• [ ] Desktop ESR uplift request, when relevant
• [ ] m-c beta or ESR uplift request, when relevant
• [ ] Intent to Ship email is sent to release-drivers@ mailing list for high priority releases
• [ ] WebCompat QA marks each Bugzilla bug associated with the release as VERIFIED

Checklist for follow-ups after landing/deploying everywhere

When the new addon version is released everywhere and there will be no more changes to that version, there are a couple of follow-up steps to do in preparation for the next release cycle.

• [ ] Tag the version on GitHub with `git tag -a v8.0.0 -m 'v8.0.0'` and push the tags with `git push --tags` so we have a persistent record of "what shipped when" on GitHub as well.

Interventions Release Rotations

Engineers on the Web Compatibility team will rotate on ownership of shipping new versions of our interventions addons, serving as an Intervention Release Owner (IRO). The process will follow a predictable 4 week schedule, mirroring the proposed 4 week Firefox release schedule, however, we will attempt to land the interventions 1 week before soft freeze (see the schedule above for dates). Another way to look at this is 2 weeks before release.

A tracking bug for the next release should be filed in Bugzilla (in the Web Compatibility::Interventions component). Bugs for adding or removing interventions in the current release cycle should block this bug.

During bug diagnosis, if a site is identified as a low priority intervention candidate, a label is added for the next IRO to take care of during their rotation. Low priority interventions ride the trains without any need for uplifts or out of band shipping mechanisms. The expectation is that there will be a single regular low-priority release for each version of Firefox, driven by the IRO.

High priority interventions should be flagged immediately to the IRO who will then begin the process necessary to ship an off-train intervention.

Interventions Release Rotations

Engineers on the Web Compatibility team will rotate on ownership of shipping new versions of our interventions addons, serving as an Intervention Release Owner (IRO). The process will follow a predictable 4 week schedule, mirroring the proposed 4 week Firefox release schedule.

During bug diagnosis, if a site is identified as a low priority intervention candidate, a label shall be added for the IRO to take care of during their rotation. Low priority interventions ride the trains without any need for uplifts or out of band shipping mechanisms. The expectation is that there will be a single regular low-priority release for each version of Firefox, driven by the IRO.

High priority interventions should be flagged immediately to the IRO who will then begin the process necessary to ship an off-train intervention.

Candidates for interventions

There is a list of sites as well as a Bugzilla bug query for sites that need interventions. In addition, the `type-uaoverride` label may be useful to look at. During a rotation, you should look at both of these sites and determine which are the most important to work on and ship (or close, if appropriate).

Lists of currently deployed interventions across the products and channels are available here https://arewehotfixingthewebyet.com/.

In addition, there is also a `sitepatch-applied` label that helps to keep track of existing interventions. Take care to add this to bugs after shipping a site patch.

IRO Rotation Responsibilities

• Authoring, testing, landing high priority intervention patches
• Authoring, testing, landing normal priority interventions
• Requesting testing of high risk interventions from the WebCompat QA team
• Coordinating QA verification and release stakeholders (Balrog, AC, etc) for high priority interventions
• Sending an intent-to-ship email for high priority intervention patches
• Landing patches in GitHub, Mozilla Central, and Android Components repos
• Requesting uplifts for interventions when necessary
• Backporting interventions from mozilla-central where necessary (see Backporting a Mozilla Central Patch).
• Handling potential regression fallout from interventions
• Add sitepatch-applied labels to web-bugs issues, as relevant.

Intervention Release Owner Schedule

For simplicity, we rotate on a per-release following the Nightly schedule at [Release Calendar].

Upcoming Releases

• Firefox 97 (v30) - Ksenia 2022-01-06 2022H1

Past Releases

• Firefox 74 (v7) - Ksenia
• Firefox 75 (v8) - Dennis
• Firefox 76 (v9) - Mike
• Firefox 77 (v10) - Ksenia
• Firefox 78 (v11) - Mike
• Firefox 79 (v12) - Dennis
• Firefox 80 (v13) - Karl
• Firefox 81 (v14) - Ksenia
• Firefox 82 (v15) - Tom
• Firefox 83 (v16) - Dennis
• Firefox 84 (v17) - Karl
• Firefox 85 (v18) - Ksenia
• Firefox 86 (v19) - Tom 2021-01-21 2021Q1
• Firefox 87 (v20) - Dennis 2021-02-18 2021Q1
• Firefox 88 (v21) - Karl 2021-03-18 2021Q1
• Firefox 89 (v22) - Ksenia 2021-04-15 2021Q2
• Firefox 90 (v23) - Tom 2021-05-27 2021Q2
• Firefox 91 (v24) - Dennis 2021-07-08 2021Q2
• Firefox 92 (v25) - Karl 2021-08-05 2021H2
• Firefox 93 (v26) - Ksenia 2021-09-02 2021H2
• Firefox 94 (v27) - Tom 2021-09-30 2021H2
• Firefox 95 (v28) - Dennis 2021-10-28 2021H2
• Firefox 96 (v29) - Karl 2021-12-02 2021H2