User:Mconnor/BIDSync(s)

Project Summary

Sync is moving to use BrowserID. The Apps product also needs Sync, and introduces a hard requirement for access to data from HTML5 web apps. This is a joint project to build out a shared platform, based on the current Firefox Sync implementations and APIs.

High level concepts

• We will use BrowserID+OAuth for authentication to service endpoints.
  • This includes a token/service discovery service that points clients to resources
• BrowserID will provide a key wrapping service that will allow the Sync Key to be stored, safely, alongside of the Sync data.
• AppSync will be in most ways, just another Sync engine, but the data will be stored in a durable form, as a HA (three nines?) cluster with backups, and a to-be-defined target for durability/geo-redundancy
• We will not include backwards compatibility with v1.1 users. This is a flag day, plain and simple.

Key Personnel

Engineering
Server	Toby Elliott
Desktop Firefox	Gregory Szorc
Android Firefox	Richard Newman
HTML5 Client (Apps)	Ian Bicking
Products
Identity	Dan Mills
Apps	Ragavan Srinivasan
Management Contacts
Services	Mike Connor
Apps	Bill Walker
Services Ops	Jeff Vier

Desktop Client

BrowserID Support

People

Developer Lead	Gregory Szorc (gps)
Additional Engineers

Requirements

• Implement support for the v2.0 protocol based on BrowserID
• Hook key storage into BrowserID key wrapping/unwrapping
• Implement support for revised UX based on BrowserID login/account creation/migration from existing Sync credentials.

Risks

• UX portions are blocked on Product/UX coherency
• BrowserID key wrapping is not implemented

App Sync integration

People

Developer Lead	Anant N (anant)
Additional Engineers

Requirements

• Define and document app-neutral data format for Apps metadata (shared with Android/HTML5 implementations)
• Write a Sync engine for Apps using the shared data format

Android Client

BrowserID Support

People

Developer Lead	Richard Newman (rnewman)
Additional Engineers	Harald Kirschner (digitarald), Chenxia Liu (liuche)

Requirements

• Implement support for the v2.0 protocol based on BrowserID
• Hook key storage into BrowserID key wrapping/unwrapping
• Implement support for revised UX based on BrowserID login/account creation/migration from existing Sync credentials.

Work Breakdown

This falls into three main parts: setup UX, request authentication, and key handling.

Request authentication

• Build new assertion-based authentication mechanism for HTTP requests.
  • Modify CredentialsSource, BaseResource, SyncAdapter, ….
  • Automated tests.
  • Implement and test “401 → invalidate token and abort” flow.

Key and credentials handling

• Alter SyncAuthenticatorService to fetch and return BrowserID assertions instead of credentials.
  • Solve "missing Extras" problem that currently requires token invalidation on each sync.
  • Consider impact on design of possibly providing system-wide BrowserID.
  • SyncAuthenticatorService will now make HTTP requests….
• Implement key wrapping and Sync Key fetching functionality.
  • What to do if one of the keys appears to be wrong?
• Credentials migration.
  • Does Android Sync have to wrap and upload your key if it gets there first?!
• Privacy and sec review.
• Thorough tests.

UX

• Alter icons and strings.
• Implement BrowserID login screens, error handling UI, help links.
  • Fallback to non-key-storage version.
  • J-PAKE integration for people who don't like typing.
• Write setup and support docs for SUMO.
• QA scripts.

Target Timeline with made-up numbers

• Protocol and keys coding unlikely to begin in earnest until March, due to lack of test infrastructure and other commitments. UI work can start as soon as designs are firm.
• Authenticator work and HTTP handling: ~2 weeks.
• Basic key fetching/unwrapping/verification etc.: ~2 weeks, partly in parallel.
• UI work… no idea. Depends on the UI design!
• Robustness, testing, baking: ~3 weeks, or as long as we have.
• In summary: this will be very tight. The sooner infrastructure is available against which to begin testing, the more likely we are to succeed.

Risks

• UX portions are blocked on Product/UX coherency.
• BrowserID key wrapping is not implemented.
• Most work can't begin until a Sync 2.0 protocol endpoint and the assorted key wrapping services are available to code against, because coding against a spec Doesn't Work®.
• Existing massive Android Sync work backlog, along with Aurora and Beta support.
• Inadequate time for sec review.
• rnewman potential unavailability.

App Sync integration

People

Developer Lead	Harald Kirschner (digitarald)
Additional Engineers

Requirements

• Define and document app-neutral data format for Apps metadata (shared with Desktop Client)
• Create an Android content provider for access to Apps local store
• Write a Sync repository for Apps using the shared data format and the content provider

HTML5 Client

People

Developer Lead	Ian Bicking (ianbicking
Additional Engineers	TBD

Requirements

• Identify what requirements the HTML client has for the server
• Implement/adapt sync client
• Develop testing routine across supported clients/browsers

Risks

• BrowserID key wrapping not implemented.
• We need an app serialization format to move forward on the implementation.
• HTML clients have more restricted cross-origin requirements; facilitating this kind of access requires server support, maybe a proxy.
• HTML clients might not support crypto efficiently (still not sure if we're doing crypto or not).

Server Work

BrowserID key wrapping and master key storage

People

Developer Lead	Ben Adida (benadida)
Additional Engineers	TBD

Requirements

• Provide an API for BrowserID consumers to have a key bundle wrapped/unwrapped using a master key
• Store master key with BrowserID provider

Risks

• crypto is hard
• work not yet started

Discovery Service

People

Developer Lead	Tarek Ziade (tarek)
Additional Engineers	Alexis Metaireau (alexis)

Requirements

• Provide a file that represents an entry point into a Sagrada system. Can be static.

Risks

• File format needs defining. It's not going to be complex, though.

Token Server

People

Developer Lead	Tarek Ziade (tarek)
Additional Engineers	Alexis Metaireau (alexis)

Requirements

• Define flow for retrieving and converting tokens to be used in other Sagrada apps
• Implement HA, scalable server with entrypoint for each service
• Do node-assignment to large-scale installations to distribute users across the instance

Risks

• High-volume crypto at scale is tricky

Storage API v2.0

People

Developer Lead	Ryan Kelly (rfkelly)
Additional Engineers	Tarek Ziade (tarek) (as consultant on current system)

Requirements

• Support consumption of Sagrada tokens as primary method of authentication
• Use Sagrada metadata tokens to identify backend storage location
• Document and implement changes to 2.0 API (almost all unused feature removal)
• Take the clean-room opportunity to fix some long-standing database problems

Work Breakdown

Design and Document the 2.0 Protocol API

• This is mostly done already, as it's just some cleanup and removal of unused features.
• The API may change a little based on client developer feedback..?
• Estimate: 1 day, then tweaking as needed in parallel with other tasks.

Migrate server-storage to pyramid/cornice/mozsvc

• This involves replacing the use of server-core with our new pyramid-based libraries.
• Based on account-portal experience, the backend code can remain pretty much unchanged.
• It should be possible to pass almost all of the existing test suite with the new core.
• No protocol changes implemented yet, one thing at a time!
• Estimated Time to Testability: 2 days.
• Then testing/bugfixing as other work progresses.

Consume Sagrada Tokens

• In theory this means just plugging in the "token processing library" described below.
• Current token design includes no metadata, just the UID - so do we still need to look up the other details in LDAP?
• Estimated Time To Testability: 1 day, assuming token processing library is ready and working...

Implement the 2.0 Protocol API

• Some of the changes we get for free by using cornice.
• Others will be small self-contained amounts of work, e.g. rename headers, change urls, etc.
• Needs new logging infrastructure, since URL/header changes make zeus logs unusable for metrics.
• Estimated Time To Testability: 2 days.

Deploy for Client Testing

• At this point, clients can start developing/testing against the new protocol.
• We should get something up and running for them to test against. A local server? A dev VM?

Implement new database backend

• Purely internal, clients should see no changes at this point.
• Heavily based on existing SQL backend, but with some simplifications.
• Need to coordinate with ops to update DB management scripts.
• Also need to investigate impact on the memcached layer.
• Estimated Time To Testability: 3 days.

Fixing other Known Bugs

• Once the necessary changes are in place, we should focus on existing server bugs.
• Particularly any that will help reduce problems during the launch:

   * database connection closing/timeout improvements
   * defering of deletes to a background process
   * quota management
   * others...?

Risks

• Because users will reupload all their stuff, FF launch day is likely to be a heavy, heavy data day.
• The use of a separate metadata token is currently undefined, it's not clear how or when that will fit into this plan.

Metrics (metlog)

People

Developer Lead	Rob Miller (RaFromBRC)
Additional Engineers	Victor Ng (vng)

Requirements

• Produce internal library for centralized logging
• Process incoming logging data along entire pipeline to final storage location(s)
• Some initial graph interpretations of incoming data

Risks

• While this is nice to have and probably achievable, it is not strictly a blocker to BID sync

Token Processing Library

People

Developer Lead	Ryan Kelly (rfkelly)
Additional Engineers	Tarek Ziade (tarek)

Requirements

• Abstract out the code that turns a Sagrada token into a userid and some additional metadata into a library that all Sagrada services can use.

Durable Node Cluster

People

Developer Lead	Richard Soderberg (atoll)
Additional Engineers	Pete Fritchman (petef)

Requirements

• Operational deployment of v2.0 webheads, talking to a Replicated MySQL cluster with backups
• Durability of data is a requirement. HA requirement is similar to sync.

Security Assessments

• Threat/security analysis (yvan)