Security/FirefoxOperations
Cloud Services Security
The CloudSec team is tasked with securing core Firefox services operated by the Cloud Services organization at Mozilla.
Contact
Email us at cloudsec@mozilla.com with the PGP key Mozilla Cloud Services Security (CloudSec) 6F73539153B31C193A2154EAF7A9B793541A953D
To report a security issue on a given site, use the bug bounty form as explained here.
Backlog
The table below summarizes the open issues assigned to the CloudSec team, sorted by area of focus.
Operational Security
Continuous Testing (TDS) |
Fraud Detection |
User management |
Infra Hardening |
Threat monitoring |
---|---|---|---|---|
Application Security
Risk & Security reviews |
Test & Implement Baseline Security |
Data & Code Signing |
Training & Communication |
Bug Bounty |
External audits |
---|---|---|---|---|---|
no pending task |
Strategy
1. Improve operational security of the core infrastructure
1.1 Implement Test Driven Security (TDS) in CI/CD
Security tests should be part of the continuous integration (CI) and continuous delivery (CD) pipelines.
- CI integration should be part of the code commit/review process, either in an existing CI (travis-ci, circleci, taskcluster) or in a security dedicated one. CI tests should include static code analysis and recommendations, docker containers testing and dependency checks (vulnerability management).
- CD integration should be done at Jenkins' level, when stage environments are built and promoted.All services are regularly rebuilt by Jenkins. CD tests should include application vulnerability scanning (ZAProxy) and infrastructure access control tests (security groups, IAM permissions, ...).
TDS should output directly in the build pipeline at first, and allow dev & ops to control levels that block integration & delivery. In a second phase, TDS outputs should be aggregated into a central security tracking platform.
1.2 Make use of the logging pipeline to detect fraud and anomalies
Heka, ElasticSearch and Kafka are powerful tools on top of which we can plug various pattern detection mechanisms to identify known bad actors, or unusual behavior. Fraud detection is a highly requested feature that devs don’t want to rebuild every time. Fraud detection should operate autonomously for each service, taking into account business rules set by the developers and the security team.
1.3 Improve user management and authentication
We should make better use of LDAP to add and remove employees from various third party services and admin panels.
- Admin panels should rely on Mozilla's Identity Management platform provided by IT
- Third-party services (datadog, pagerduty, aws) should have automated user management (userplex).
Cloudsec need to facilitate integration with Mozilla's IAM with standard libraries and tools.
1.4 Harden the infrastructure
All services and tools that are part of the standard infrastructure should undergo security hardening. Hardening rules should be testable in the CD pipeline (see TDS above) to prevent security regressions. Some examples:
- SSH should enforce MFA authentication
- Disabled users should be removed from all systems, particularly bastion hosts
- AWS permissions must prevent services from compromising each other
- Secrets must be provisioned encrypted
- ...
2. Increase security maturity
2.1 Help new projects identify threats and controls (RRA, threat models,...)
Risk assessment and threat modeling help people think through failure scenarios they wouldn’t evaluate otherwise. RRAs often leads to architectural changes that are best identified early. Each new project must undergo a 30/60min RRA with one of the member of cloudsec to assess the security posture of the project.
2.2 Implement baseline services security standards
Content Security Policy (CSP), HSTS, HPKP, data signature and encryption, input validation, XSS and SQLi protection are part of techniques developers should care about when building new services. Cloudsec defines services security standards that devs can implement and cloudsec tests in TDS.
2.3 Communicate security effectively throughout the organization
Teams need a channel to ask security questions, discuss concerns and share techniques. CloudSec must organize information flow and broadcast to developers, ops and managers. This includes general security best practices, analyzis and actions to take on CVE vulnerabilities, response and communication on incidents.
2.4 Use Mozilla’s bug bounty program
The bug bounty program is a fantastic tool: for a small amount of money, we reward people worldwide for helping us improve our security posture. Most security issues identified in our services come from the bug bounty program. We must ensure that all services are part of the bug bounty program and that triaging is performed regularly. As much as possible, we must assist developers in fixing security issues that are reported through bug bounties.
3. Build core security services
3.1 Sign data that changes the configuration of user agents
We iterate fast, and eventually someone, either us or a partner, is bound to make a mistake and open a door that could put our users at risk. Signing the data we send to our users helps cover that risk. Digital signature for Firefox is a complex topic - not every project can implement it independently - so cloudsec must provide the tooling and services to facilitate signing ([autograph](https://github.com/mozilla-services/autograph))
3.2 Monitor our ecosystem for external threats
There are many scenarios in which our users can be at risk because of the fraudulent or careless behavior of a third party. A bad certificate authority could issue a certificate that impersonates us. A careless partner could leak addon signing keys. A web startup could get hacked and leak web push endpoints. We should implement the tools needed to identify fraudulent behavior outside of our organization that impact us, so we can react in a timely manner and protect Firefox users.
3.3 Partner with external firms to monitor our security
We can’t do everything ourselves. External security firms can help us keep an eye on and audit our services. Some of their work may be redundant with current efforts, such as automated security testing, but would help cover the interim. We should evaluate various vendors and partner with the ones that have the best support of our technologies.
Security Checklist
All services integrated with Firefox or that provide services to Firefox users must follow the security rules listed below.
Infrastructure rules
- Use modern TLS (INFRA-TLS)
- Set HSTS to 2592000 (30 days) (INFRA-HSTS)
- Set HPKP to 2592000 (30 days) (INFRA-HPKP)
- Admin panels must only be available behind VPN and require LDAP auth (INFRA-ADMIN)
Coding rules
The following rules apply to all web applications: api and websites.
- Detailed logging in mozlog format (APP-MOZLOG)
- Business logic must be logged with app specific codes (errno)
- Access control failures must be logged at WARN level
- All SQL queries must be parameterized, not concatenated (APP-SQL)
- User data must be escaped for the right context prior to reflecting it (APP-ESCAPE)
- Apply sensible limits to user inputs, see Input validation (APP-INPUTVAL)
- Enforce Access Controls server-side (APP-ACL)
- Set the Secure flag on Cookies, and use sensible Expiration and HTTPOnly (APP-SECCOOKIE)
- Keep 3rd-party libraries up to date (APP-DEPS)
- Use NSP or Greenkeeper for NodeJS applications
- Use pip --outdated or requires.io for Python applications
- When handling cryptographic keys, must have a mechanism to handle monthly key rotations (APP-KEYROT)
For websites
The following coding rules only apply to websites.
- Never store passwords, use Firefox Accounts (APP-IDP)
- Forbid Mixed content, always use HTTPS (APP-MIXCONTENT)
- Must have a CSP with (APP-CSP)
- a report-uri pointing to the service /__cspreport__
- frame-options set to deny
- no use of unsafe-inline or unsafe-eval
- Must have CSRF tokens and manually excluded specific forms (APP-CSRF)
- Must have checksums for 3rd-party content via SRI (APP-SRI)
- Consider Security headers as appropriate (APP-HEADERS)
- X-Content-Type-Options
- X-Frame-Options
- X-XSS-Protection
Data rules
- Sensitive user data (like browsing history) stored on our servers must either be:
- Anonymized (similar to Tiles) (DATA-ANON)
- Encrypted client-side (similar to Sync) (DATA-CRYPT)
- Must sign data and code pushed to Firefox clients (DATA-SIGN)
- Addons must use standard AMO signing (APP-SIGNING)
- Code & Conf must use Content-Signature via Autograph (DATA-SIGNING)
- Must implement a sign-off protocol when changing data/code to the beta and release channels, see Firefox Continuous Delivery Sign Off
Sites and Services
CloudSec is responsible for the security of the following websites and backend services.
(note: cloudsec is not responsible for the security of implementations in firefox, only of the backend services).
ABSearch
Code: absearch
Public Endpoints:
- search.services.mozilla.com
In Bounty Scope? Yes
Addons.mozilla.org
Code:
Public Endpoints:
- addon.mozilla.org
- addons.mozilla.org
- blocklist.addons.mozilla.org
- builder.addons.mozilla.org
- controller-review.apk.firefox.com
- controller.apk.firefox.com
- services.addons.mozilla.org
- static.addons.mozilla.net
- versioncheck-bg.addons.mozilla.org
- versioncheck.addons.mozilla.org
In Bounty Scope? Yes
Product Delivery
Code: go-bouncer
Public Endpoints:
- download-installer.cdn.mozilla.net
- download.mozilla.org
In Bounty Scope? Yes
AUS/Balrog
Code: balrog
Public Endpoints:
- aus3.mozilla.org
- aus4.mozilla.org
- aus5.mozilla.org
- aus.mozilla.org
In Bounty Scope? Yes
Firefox Accounts
Code:
Public Endpoints:
- accounts.firefox.com
- api.accounts.firefox.com
- oauth.accounts.firefox.com
- profile.accounts.firefox.com
- verifier.accounts.firefox.com
In Bounty Scope? Yes
Firefox Sync
Code:
Public Endpoints:
- *.$region.sync.services.mozilla.com
- token.services.mozilla.com
In Bounty Scope? Yes
Firefox Hello
Code: loop-server
Public Endpoints:
- hello.firefox.com
- loop.services.mozilla.com
In Bounty Scope? Yes
Location (MLS)
Code:
Public Endpoints:
- location.services.mozilla.com
- location-leaderboard.services.mozilla.com
In Bounty Scope? Yes
Marketplace.firefox.com
Code: zamboni
Public Endpoints:
- marketplace.firefox.com
- receiptcheck.marketplace.firefox.com
- static.marketplace.firefox.com
In Bounty Scope? Yes
Persona
Code: persona
Public Endpoints:
- browserid.org
- firefoxos.persona.org
- persona.org
- static.login.persona.org
- verifier.login.persona.org
- www.browserid.org
- www.persona.org
- yahoo.login.persona.org
- gmail.login.persona.org
- login.anosrep.org
- login.mozilla.org
- login.persona.org
- diresworb.org
In Bounty Scope? Yes
Push
Code:
Public Endpoints:
- push.services.mozilla.com
- updates.push.services.mozilla.com
In Bounty Scope? Yes
Security Settings (Kinto)
Code: TBD
Public Endpoints:
- settings.services.mozilla.com
In Bounty Scope? No
Shield / Normandy
Code:
Public Endpoints: TBD
In Bounty Scope? No
Telemetry
Code:
Public Endpoints:
- incoming.telemetry.mozilla.org
- telemetry-experiment.cdn.mozilla.net
- analysis.telemetry.mozilla.org
- sql.telemetry.mozilla.org
- metrics.services.mozilla.com
In Bounty Scope? Yes
Test Pilot
Code: testpilot
Public Endpoints:
In Bounty Scope? Yes
Tiles
Code: splice
Public Endpoints:
- tiles.cdn.mozilla.net
- tiles.services.mozilla.com
In Bounty Scope? Yes
TLS Observatory
Code: tls-observatory
Public Endpoints:
- tls-observatory.services.mozilla.com
In Bounty Scope? No
Tracking Protection
Code: shavar
Public Endpoints:
- shavar.services.mozilla.com
- tracking.services.mozilla.com
In Bounty Scope? Yes
Everything.me
In Bounty Scope? No
Find My Device
Code: find my device
In Bounty Scope? No