Release Management/Chemspill

Definition

"Chemspill" is a term used to describe a security-driven rapid release.

In a "chemspill" situation we release on whichever channels necessary, with only the necessary patch(es), as fast as possible. This is usually reserved for situations where a critical security exploit is public.

Some documentation around chemspill process

• Release days for Dot releases and chemspills
• Chemspill process description. A template to copy and use for organizing incident response.
• Chemspill retrospective template. Use this for post-mortems (in draft, June 2019)
• Slides from a relman lightning talk on a chemspill in 2018

Past chemspills

2019 Jun "Coinbase hack"

2 chemspills during all hands work week.

• Versions:
  • Jun 18 - 67.0.3, 60.7.1esr, Firefox for Android 67.0.3. Fixed type confusion in Array.pop.
  • Jun 20 - 67.0.4, 60.7.2esr, Firefox for Android 67.0.4. Fixed a sandbox escape using Prompt:Open.
• Bug(s): bug 1559845; bug 1544386; bug 1559858
• Notes: Incident doc; retrospective

2019 May "Armagadd-on 2"

Not a security breach but a rapid and focused single-issue dot release, which we treated as a chemspill in some ways. Repaired certificate chain to re-enable web extensions that had been disabled.

• Versions: 66.0.4, 60.6.2esr, 67.0b17
• Bug(s): bug 1548973
• Notes: Incident doc; Technical report; ESR's Mozilla Hacks post

pwn2own 2019

IonMonkey/JIT issues

• 66.0.1, 60.6.1esr, 67.0b4
• Bugs: bug 1537924, bug 1538006
• Notes: Incident doc, retrospective

pwn2own 2018 Mar 15

Out of bounds memory write while processing Vorbis audio data.

• Versions: 59.0.1, Firefox ESR 52.7.2,
• Bugs: bug 1446062, bug 1446365
• Notes: Incident doc - Mozilla Hacks post on this chemspill

2018 Jan: Spectre/Meltdown

• Versions: 58.0.1 , 57.0.4.
• Bug(s):
• Notes: incident doc

2017 Dec: tab crash issue

Not quite a chemspill but was treated as such

• Versions: 57.0.3,
• Bug(s): Bug 1424373
• Notes: incident doc

2017 Mar, pwn2own

• Versions: 52.0.1
• Bug(s): Bug 1348168
• Notes:

2016 Nov 30, SVG 0day

• Versions: 50.0.2, 51.0b5, and 45.5.1esr.
• Bug(s): Bug 1321066
• Notes:

2016 , "Armagadd-on"

• Versions:
• Bug(s):
• Notes: https://public.etherpad-mozilla.org/p/bug-1267318

Feb 2016 Service workers issue

• Versions: 44.0.2
• Bug(s): 1245724
• Notes:

Aug 2015, Graphite2

• Versions: ESR 38
• Bug:
• Notes:

Aug 2015, pdf.js issue

• Versions: 39.0.3, 38.1.1
• Bug(s): 1191284
• Notes:

Apr 2015

• Versions: 39.0.3.
• Bug(s):
• Notes:

Mar 2015

• Versions: 36.0.3/36.04 and 31.5.2/31.5.3
• Bugs: 1144988, 1145870
• Notes: (these were at https://etherpad.mozilla.org/36-0-chemspill-Post-Mortem)