Project Summary
Sync is moving to use BrowserID. The Apps product also needs Sync, and introduces a hard requirement for access to data from HTML5 web apps. This is a joint project to build out a shared platform, based on the current Firefox Sync implementations and APIs.
High level concepts
- We will use BrowserID+OAuth for authentication to service endpoints.
- This includes a token/service discovery service that points clients to resources
- BrowserID will provide a key wrapping service that will allow the Sync Key to be stored, safely, alongside of the Sync data.
- AppSync will be in most ways, just another Sync engine, but the data will be stored in a durable form, as a HA (three nines?) cluster with backups, and a to-be-defined target for durability/geo-redundancy
- We will not include backwards compatibility with v1.1 users. This is a flag day, plain and simple.
Key Personnel
| Engineering
|
| Server
|
Toby Elliott
|
| Desktop Firefox
|
Gregory Szorc
|
| Android Firefox
|
Richard Newman
|
| HTML5 Client (Apps)
|
Ian Bicking
|
| Products
|
| Identity
|
Dan Mills
|
| Apps
|
Ragavan Srinivasan
|
| Management Contacts
|
| Services
|
Mike Connor
|
| Apps
|
Bill Walker
|
| Services Ops
|
Jeff Vier
|
Desktop Client
BrowserID Support
People
| Developer Lead
|
Gregory Szorc (gps)
|
| Additional Engineers
|
|
Requirements
- Implement support for the v2.0 protocol based on BrowserID
- Hook key storage into BrowserID key wrapping/unwrapping
- Implement support for revised UX based on BrowserID login/account creation/migration from existing Sync credentials.
Work Breakdown
- Non-auth changes to support 2.0 protocol
- Assuming no backwards compatibility with 1.1 protocol
- xpcshell test changes will consume a lot of time since there are a lot of manually configured HTTP servers
- HTTP changes to support new auth
- service.js support for new auth methods
- UI changes for new auth
Target Timeline
Risks
- UX portions are blocked on Product/UX coherency
- BrowserID key wrapping is not implemented
- Who is writing BrowserID JSM for Firefox?
- Need actual Sync Server implementation to verify integration
App Sync integration
People
| Developer Lead
|
Anant N (anant)
|
| Additional Engineers
|
|
Requirements
- Define and document app-neutral data format for Apps metadata (shared with Android/HTML5 implementations)
- Write a Sync engine for Apps using the shared data format
Target Timeline
Risks
Android Client
BrowserID Support
People
| Developer Lead
|
Richard Newman (rnewman)
|
| Additional Engineers
|
Harald Kirschner (digitarald), Chenxia Liu (liuche)
|
Requirements
- Implement support for the v2.0 protocol based on BrowserID
- Hook key storage into BrowserID key wrapping/unwrapping
- Implement support for revised UX based on BrowserID login/account creation/migration from existing Sync credentials.
Work Breakdown
This falls into three main parts: setup UX, request authentication, and key handling.
Request authentication
- Build new assertion-based authentication mechanism for HTTP requests.
- Modify CredentialsSource, BaseResource, SyncAdapter, ….
- Automated tests.
- Implement and test “401 → invalidate token and abort” flow.
Key and credentials handling
- Alter SyncAuthenticatorService to fetch and return BrowserID assertions instead of credentials.
- Solve "missing Extras" problem that currently requires token invalidation on each sync.
- Consider impact on design of possibly providing system-wide BrowserID.
- SyncAuthenticatorService will now make HTTP requests….
- Implement key wrapping and Sync Key fetching functionality.
- What to do if one of the keys appears to be wrong?
- Credentials migration.
- Does Android Sync have to wrap and upload your key if it gets there first?!
- Privacy and sec review.
- Thorough tests.
UX
- Alter icons and strings.
- Implement BrowserID login screens, error handling UI, help links.
- Fallback to non-key-storage version.
- J-PAKE integration for people who don't like typing.
- Write setup and support docs for SUMO.
- QA scripts.
Target Timeline with made-up numbers
- Protocol and keys coding unlikely to begin in earnest until March, due to lack of test infrastructure and other commitments. UI work can start as soon as designs are firm.
- Authenticator work and HTTP handling: ~2 weeks.
- Basic key fetching/unwrapping/verification etc.: ~2 weeks, partly in parallel.
- UI work… no idea. Depends on the UI design!
- Robustness, testing, baking: ~3 weeks, or as long as we have.
- In summary: this will be very tight. The sooner infrastructure is available against which to begin testing, the more likely we are to succeed.
Risks
- UX portions are blocked on Product/UX coherency.
- BrowserID key wrapping is not implemented.
- Most work can't begin until a Sync 2.0 protocol endpoint and the assorted key wrapping services are available to code against, because coding against a spec Doesn't Work®.
- Existing massive Android Sync work backlog, along with Aurora and Beta support.
- Inadequate time for sec review.
- rnewman potential unavailability.
App Sync integration
People
| Developer Lead
|
Harald Kirschner (digitarald)
|
| Additional Engineers
|
|
Requirements
- Define and document app-neutral data format for Apps metadata (shared with Desktop Client)
- Create an Android content provider for access to Apps local store
- Write a Sync repository for Apps using the shared data format and the content provider
Target Timeline
Risks
HTML5 Client
People
| Developer Lead
|
Ian Bicking (ianbicking
|
| Additional Engineers
|
TBD
|
Requirements
- Identify what requirements the HTML client has for the server
- Implement/adapt sync client
- Develop testing routine across supported clients/browsers
Target Timeline
Risks
- BrowserID key wrapping not implemented.
- We need an app serialization format to move forward on the implementation.
- HTML clients have more restricted cross-origin requirements; facilitating this kind of access requires server support, maybe a proxy.
- HTML clients might not support crypto efficiently (still not sure if we're doing crypto or not).
Server Work
BrowserID key wrapping and master key storage
People
| Developer Lead
|
Ben Adida (benadida)
|
| Additional Engineers
|
TBD
|
Requirements
- Provide an API for BrowserID consumers to have a key bundle wrapped/unwrapped using a master key
- Store master key with BrowserID provider
Target Timeline
Risks
- crypto is hard
- work not yet started
Discovery Service
People
| Developer Lead
|
Tarek Ziade (tarek)
|
| Additional Engineers
|
Alexis Metaireau (alexis)
|
Requirements
- Provide a file that represents an entry point into a Sagrada system. Can be static.
Target Timeline
Risks
- File format needs defining. It's not going to be complex, though.
Token Server
People
| Developer Lead
|
Tarek Ziade (tarek)
|
| Additional Engineers
|
Alexis Metaireau (alexis)
|
Requirements
- Define flow for retrieving and converting tokens to be used in other Sagrada apps
- Implement HA, scalable server with entrypoint for each service
- Do node-assignment to large-scale installations to distribute users across the instance
Target Timeline
Risks
- High-volume crypto at scale is tricky
Storage API v2.0
People
| Developer Lead
|
Ryan Kelly (rfkelly)
|
| Additional Engineers
|
Tarek Ziade (tarek) (as consultant on current system)
|
Requirements
- Support consumption of Sagrada tokens as primary method of authentication
- Use Sagrada metadata tokens to identify backend storage location
- Document and implement changes to 2.0 API (almost all unused feature removal)
- Take the clean-room opportunity to fix some long-standing database problems
Work Breakdown
Design and Document the 2.0 Protocol API
- This is mostly done already, as it's just some cleanup and removal of unused features.
- The API may change a little based on client developer feedback..?
- Estimate: 1 day, then tweaking as needed in parallel with other tasks.
Migrate server-storage to pyramid/cornice/mozsvc
- This involves replacing the use of server-core with our new pyramid-based libraries.
- Based on account-portal experience, the backend code can remain pretty much unchanged.
- It should be possible to pass almost all of the existing test suite with the new core.
- No protocol changes implemented yet, one thing at a time!
- Estimated Time to Testability: 2 days.
- Then testing/bugfixing as other work progresses.
Consume Sagrada Tokens
- In theory this means just plugging in the "token processing library" described below.
- Current token design includes no metadata, just the UID - so do we still need to look up the other details in LDAP?
- Estimated Time To Testability: 1 day, assuming token processing library is ready and working...
Implement the 2.0 Protocol API
- Some of the changes we get for free by using cornice.
- Others will be small self-contained amounts of work, e.g. rename headers, change urls, etc.
- Needs new logging infrastructure, since URL/header changes make zeus logs unusable for metrics.
- Estimated Time To Testability: 3 days.
Deploy for Client Testing
- At this point, clients can start developing/testing against the new protocol.
- We should get something up and running for them to test against. A local server? A dev VM?
Implement new database backend
- Purely internal, clients should see no changes at this point.
- Heavily based on existing SQL backend, but with some simplifications.
- Need to coordinate with ops to update DB management scripts.
- Also need to investigate impact on the memcached layer.
- Estimated Time To Testability: 3 days.
Fixing other Known Bugs
- Once the necessary changes are in place, we should focus on existing server bugs.
- Particularly any that will help reduce problems during the launch:
* database connection closing/timeout improvements
* defering of deletes to a background process
* quota management
* others...?
Target Timeline
Risks
- Because users will reupload all their stuff, FF launch day is likely to be a heavy, heavy data day.
- The use of a separate metadata token is currently undefined, it's not clear how or when that will fit into this plan.
Metrics (metlog)
People
| Developer Lead
|
Rob Miller (RaFromBRC)
|
| Additional Engineers
|
Victor Ng (vng)
|
Requirements
- Produce internal library for centralized logging
- Process incoming logging data along entire pipeline to final storage location(s)
- Some initial graph interpretations of incoming data
Target Timeline
Risks
- While this is nice to have and probably achievable, it is not strictly a blocker to BID sync
Token Processing Library
People
| Developer Lead
|
Ryan Kelly (rfkelly)
|
| Additional Engineers
|
Tarek Ziade (tarek)
|
Requirements
- Abstract out the code that turns a Sagrada token into a userid and some additional metadata into a library that all Sagrada services can use.
Work Breakdown
Finalize repoze.who.plugins.vepauth
- much of the protocol flow has already been implemented as a standalone repoze.who plugin
- it just needs some tweaking as the details all get finalized
- (this also has basic token-provisioning support, which might be handy for testing purposes until the real tokenserver comes online)
Code to manage per-node secrets
- need code to parse the token-server secrets file
- also to watch it for changes and update secrets automatically?
Migrate mozsvc.user to use vepauth by default
- we already have code to automate authentication against a repoze.who plugin and expose the result as "request.user".
- it currently uses basicauth by default; make it use vepauth with appropriate settings
- provide extra utility functions? Need feedback from other projects on what is required of the API.
Target Timeline
Risks
Durable Node Cluster
People
| Developer Lead
|
Richard Soderberg (atoll)
|
| Additional Engineers
|
Pete Fritchman (petef)
|
Requirements
- Operational deployment of v2.0 webheads, talking to a Replicated MySQL cluster with backups
- Durability of data is a requirement. HA requirement is similar to sync.
Target Timeline
Risks
Security Assessments
- Threat/security analysis (yvan)