Security/Reviews/Firefox/Add-on hotfix
- Items to be reviewed
- https://wiki.mozilla.org/Features/Desktop/Add-on_hotfix | Feature Page
Introduce Feature
Goal of Feature, what is trying to be achieved (problem solved, use cases, etc)
- Provide a low-touch way to update Firefox users in the field without a "full" software update
- The add-on would hopefully replace the need for #.0.1 releases and we'd be able to respond to security issues even more quickly
- Ship security and chemspill-type fixes faster (no recompile required), users download them faster (smaller download), and get them installed faster (no restart required)
- Avoiding recompile is especially helpful given the use of PGO for binaries (PGO increases testing requirements and makes binary updates huge)
- Mitigation for problems with app update (whether the problems affect 0.01% or 90% of users)
- We aren't yet sure what other cases we'll use add-on hotfixes.
What solutions/approaches were considered other than the proposed solution?
Different update snippets / mars
"Special" kind of add-on
One-off system to download a .tgz from mozilla.org
Why was this solution chosen?
Minimal code changes
Using sec-reviewed systems and code
AMO
Add-on manager
Add-on system
Any security threats already considered in the design and why?=
Users can disable the add-on, 'perhaps delaying security fixes
This concern is no different than normal updates
I disagree. We're always telling users "Try disabling all your addons" and "Disable unncessary addons". I'd prefer for this addon to not be listed in about:addons.
Sometimes these addons will simply run, make some config settings, and then uninstall themselves. In the recent cases where we might have used this mechanism that would have been the way we did it.
How will AMO know not to give you the addon again?
Users who choose an addon server other than addons.mozilla.org
There's a separate pref for "magic hotfix server" and "addon update server" :)
Threat Brainstorming
When we use hotfixes to fix security holes:
Not all users get the update? So we'd have to ship a binary fix too.
We exclude this addon from the "[x] Update my add-ons" pref :)
If we ship a hotfix, and the next day someone downloads a binary from mozilla.org, are they vulnerable for the first few hours?
The extra complexity (e.g. "what happens if you install the hotfix and then the fixed version") may make it more likely to introduce regressions, including security regressions. But this is a case-by-case question, not a question about the feature.
With strict-compat warnings, the hotfix addon could slow down Firefox update.
Maybe we'll exclude the hotfix addon from compat checks.
Or better, happily uninstall it whenever there's a Firefox update.
Who will have the ability to ship these hotfixes? How does this compare to the set of people with the ability to ship new versions of Firefox?
By using AMO, we're adding attack surface for the 20% of users who don't have any addons installed (?)
Creating a custom solution is also likely to introduce flaws that AMO does not have
Also add-on blocklist is served from AMO
A silently installed/updated addon is a different risk profile than typical (even popular) add-ons. Its becomes equivalent to our existing update mechanism, so we should ensure similar mitigations are in place (i.e. code-signing with a hardcoded root)
Our application updates are not currently signed, though that's being added as part of the Silent Update feature
Our AMO updates are currently only secured by TLS
True but our current update service has a very small attack surface, unlike AMO. If we only rely on the integrity of AMO we have introduced the risk that compromise of 1 addon affects 100% of our users.
Binary addons - could complicate things, introduces code-level issues, etc - initially we are planning to ship JS, restartless addons
What will happen for users who have chosen "Check for updates, but let me choose whether to install them"? (Don't want to scare paranoid users away from having check-for-updates enabled.)
For now, those users won't get hotfixes at all :( Need UI design if we want to honor the pref and prompt for hotfixes.
Some users have chosen that pref because they want to know about major updates, but are fine with minor updates (including hotfixes) being fully automatic.
maybe trigger check/install when users do a manual app update check?
Conclusions / Action Items
{before ship}
- [clegnitto] Add to the checklist for making hotfixes: “Each specific hotfix update would need careful scrutiny (QA, security review)”
- [mossop] Hotfix addon must be authenticated additionally by code signing cert with fingerprint hardcoded in the release.
- [Jesse] argue more about "ask first" pref
find out what % have updates set to "ask first" (and currently would not get these hotfixes). If it's large may need to add UI for prompting (door hanger?), or simply ignore the "ask first" pref?
or treat it according to the blocklist pref?
or add a separate option to disable hotfixes?
[jesse] file a bug with [sg:want?]. the bug won't block the feature.
- [mossop] make sure setting the strict compat pref doesn't mean hotfixes prevent you from getting app updates
Background
Christian Legnitto wrote:
- 7.0: users would have been fixed by the very act of installing the
hotfix (no-op code-wise)
- 8.0: We could have turned off Java only for those on Mac OS X who
had installed the most recent Apple java update (no crashes with the older java).We could have then looked at the data and decided if the problem was widespread enough to spin a 8.0.1. Cool. Testing this stuff is a risk for sure....but not really appropriate for a security review :-) I think it is very relevant because it helps determine if/when we can use this mechaism to fix security vulnerabilities. We would have to ensure that sysadmins are aware of this mechanism and know exactly how to push it out to all machines so that it is active for all user profiles--including user profiles that haven't been created yet. Yeah, we'll let them know when it lands but again, this is a hotfix. We're trying to cover the 80% case. A fair amount of our user base doesn't even install the new Fx version in the 6 weeks it is current, I don't think we'd be making the status quo any worse off. This, and the responses below, make me think that this wouldn't be useable as a solution for a (0-day) critical security vulnerability. We need an enterprise-applicable solution, and one that people can manually activate and verify, for security fixes. That isn't to say that we shouldn't do it at all. Christian pointed out some good non-security-critical examples of where it would be useful. We just need to be clear on appropriate vs. inappropriate uses of this mechanism. Also, this isn't to say we couldn't make this more appropriate as a security fix mechanism in future iterations, if it would be helpful to do so after silent install is improved. - Brian [I didn't write anything below here. Just quoting so you can read what I am citing as "responses below" above.] Also, most enterprises are on 3.6 currently, so this is a concern in the future and not so much in the present. It respects update preferences and installs into the user profile (aka seen as installed by the user). This was an explicit choice to minimize risk / security impact. More suggestions: 1. "Check for updates" should force a download of this update. Again, we're sticking as close to add-on behavior as possible to minimize engineering and testing. 2. The presence of this update should affect the version information reported to the user in the about dialog box. Perhaps it should not change the version number, but there should be some notice that the security fix is installed. We thought about this and discounted it. This will generally be "silent" in that users don't need to know and websites won't either. It will show up in the add-ons list and there will be a pref set so even with a hotfix that uninstalls itself we can trace back (and prevent an install loop). It's an add-on, so if we later want to do this we can (set a pref, add to the UA, change the Fx version, etc). I see this more as policy and less critical for the Fx code. 3. Ideally, the various mechanisms we use to control addons' enablement/installation should not affect this addon. I believe right now we key off the update prefs, though the original plan was to act like a normal add-on. Dave and I discussed and agreed it made sense to key off the app update preferences instead (this is one of the only places we diverge from a regular add-on). 4. Will this addon appear in the addon manager? IMO, it would be better if it didn't. Yes. Again, specced out this way to reduce risk...in an ideal world it would appear in it's own category (l10n) and could not be uninstalled or disabled (security risk of impostor add-ons). But, we also have to weigh the risk of making these changes against the desire to have such a feature early in the development cycle. The feature was explicitly specced out for the 70-80% case, as the original estimates were that it would be ~15 lines of code, not require any more extensive testing, no supporting server-side changes, no l10n, etc. If we had a year we could make this feature awesome, polish all the rough edges, etc. Note that specced out to cover the 70-80% case is different than 70-80% done. Basically we're (ab)using the existing add-on infrastructure as a dumb pipe to shuttle bits to the majority of our users quickly and securely. I suspect a lot of the use cases will go away when we have all the silent update pieces in place.