Services/Sync/NextGen

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

Overview

Sync is a very useful feature, but has so far failed to achieve widespread adoption. We believe Sync to be an important piece of the Firefox experience, and our goal is to deliver a much more attractive product to support continued Firefox growth. We are tentatively targeting Firefox 20 as the GA for this new version of the service.

Use Cases

Needs a narrative for each

v1

• A Firefox user with multiple devices (desktop or mobile) who wants to have their awesomebar and passwords available on all of their devices.
  • Suleta uses Firefox on her Windows XP laptop and just installed Firefox on her brand new Galaxy S III smartphone. She really appreciates the Firefox awesomebar experience and Firefox's saved logins. Both of those features would be even more valuable on her smartphone where typing URLs and passwords is a real PITA. With Firefox's NextGen Sync, all she has to do is log in to her Firefox on XP (creating a Firefox account in the process) and then log in to her Firefox on Android and all of her valuable Firefox settings and data are immediately pushed to her phone.

• A Firefox user setting up a new/replacement device wants to get up and running with their full Firefox experience as fast as possible.
  • Yaholo dropped his shiny new retina Macbook Pro out of a third story window and the Geniuses at Apple gave him a brand new one. Unfortunately, they weren't able to rescue the data off of his shattered SSD so he's back to square one. With Firefox's NextGen Sync, he doesn't have to worry because he knows that as soon as he downloads the latest version of Firefox and logs in to it, all his settings and data are magically restored.

• A Firefox user who wants to back up their data easily and safely, and only has a single device.
  • Nadie lives in her Android Firefox browser. It's her connection to the outside world. She values the information that Firefox has helped her gather over the last 7 years and fears that if it were ever lost, her life would be over. Having logged in to Firefox's NextGen Sync and back up service, she surfs the web with confidence and piece of mind knowing that all of her Firefox data and settings live in a reliable Mozilla cloud instead of her local machine and that even if she loses her Firefox login credentials, she can still get her most important data and settings back.

• A Firefox user who wants to have the option of secure syncing between devices, using a strong key they manage themselves. (Parity with/comparisons to Chrome Sync, OS X Lion DiskVault).
  • Cochise, a Debian Linux Firefox user, puts security above almost all else. He would like to be able to sync his Firefox profile between his two desktop machines but he's afraid that someone somewhere will snoop on his surfing. Fortunately for Cochise, Firefox's trustworthy NextGen Sync system has an advanced option that allows him to encrypt his data so it's safe from prying eyes.

TBD

• A Firefox user who has lost their device, and their Persona password, and wants to recover their data should be able to recover everything except for passwords (and any future auth data we start syncing, such as client certificates).

• Firefox can link external services to user data on the server side.
  • Hypothetical example would be to connect del.icio.us to a user's bookmarks, and the service would connect to the server store directly, instead of requiring a client plug-in.
  • This would involve storing data in plaintext, and would almost certainly be opt-in for those users who want to use the service, possibly in the same service, or in a special-built service like AITC.

User Data covered by the feature

• Feature and defaults parity with the current Firefox Sync offering, unless there is data to support different choices.
  • Bookmarks and History (assuming that's enough to make my awesomebar go) and Passwords is Products minimum requirement.

High level technical description

Identification

The implementation of the next generation Sync feature will be centered around Persona. The Identity team will work to define and deploy a special type of IdP, to which we'll refer as a "Preferred IdP" or PIdP for short. This IdP will support an auth protocol designed to prevent disclosure of the user's Persona passphrase to the server, as well as enabling non-browser clients to directly authenticate without a browser context. In addition to a secure authentication protocol, a PIdP will also expose an API to obtain a wrapped encryption key (known as the User Key, or UK). This key will be encrypted and decrypted on the client, using a key derived from the Persona passphrase. The User Key will not be exposed to consumers, but will be used by the Persona client to encrypt and decrypt a per-service encryption key (known as a Service Key or SK) on request.

Service Authentication

As with Apps in the Cloud, the service client will obtain an assertion from the Persona client, and use that to authenticate to the Token Server. A successful request to the Token Server will return an object containing all necessary data to make requests against the Identity-attached service, as described in the Token Server API docs.

Using the Service

The new version of the Sync Storage API uses the same request authentication model as Apps in the Cloud. Beyond that, it is a significantly refined version of the 1.1 API with a set of improvements outlined in the API docs. Sync clients will use a Service Key stored on the service itself, encrypted by the Persona client using the UK. The SK (or keys chained to it) will be used to encrypt outgoing records and decrypt incoming records, like the existing model.

A very simplified view of the changes between Sync 1.1 and Sync 2.0 is provided by these diagrams:

High-level roadmap

We can split this effort into four main areas:

1. Design, build, and deploy the PIdP service (identity)
2. Design, build, and integrate the Firefox front-end to this service, on both desktop and mobile (services integration and/or Firefox frontend engineering)
   • Depends on PIdP service
3. Finish the implementation and deployment of the Sync 2.0 production service (services engineering, services operations)
4. Implement and integrate the Sync 2.0 client on desktop and mobile, including UX design and UI implementation, bearing in mind concurrent version requirements (services integration).
   • Depends on PIdP service, Firefox account support, crypto work.

Note that (2) is blocked on (1), and (4) depends on everything else to some extent. Development work can begin but not complete until those dependencies are met.

It might be helpful to think of achieving a Firefox account — that is, providing a PIdP and browser integration — as one project, and browser data synchronization as a separate project on top.

More details are in the technical plan.

Open Questions

Product Questions

• Are we still planning to ship old and new Sync products in parallel, or are we doing migration on upgrade? (Asa)
  • mconnor: This has a non-zero cost, and we're talking about 0.5% of Firefox users. We can do it, but it adds difficult-to-scope time/complexity.
  • For how long?
  • Assumption: even if shipping in parallel, not simultaneously functional. This would be an undue development and testing burden.
    • We're going to ship in parallel until we reduce the users of previous generation Sync to some minimal level at which point we'll EOL the previous version. - A

• J-PAKE as an option for users? (Asa)
  • mconnor: Not required for sure, but if we're allowing users to provide their own keys, manual entry really sucks on mobile. NFC could be a future option, making this unnecessary, but we should decide this explicitly. The cost to maintain/preserve is not dramatically high, but is of course more than zero.
    • I don't think we should do this. I know typing on some mobile devices is a pain but I don't think the J-PAKE solution conceptually less burden. - A

• Is recovery after password reset a required v1 use-case? (Asa, Karen, mconnor)
  • mconnor: Timeline/model for an escrow service is as yet undefined, adding substantial schedule risk. My recommendation is that this is a nice-to-have for v1 of the new service.
    • We don't have any recovery system now, right? If that's the case, I think we could make recovery a high priority v2 feature. We should talk about this because I don't know what's involved. - A

• What degree of durability is required for data stored on the service? (Asa, mconnor)
  • mconnor: This is going to be a budget question. It needs a cost/benefit tradeoff from mmayo's team. From prior discussions, more than 99.9999999% will be much more expensive, but 1 in a billion feels pretty okay.
  • Cost estimates and recommendations coming from Bob soonish.
    • I don't know what industry standard is. Does anyone else? I also don't know how to think about durability because at a certain point the numbers get too small/big for my little brain to process intelligently. - A

• What is the user expectation for data availability, above and beyond "the bits in the cloud don't go away"? Durability would mean we don't lose your bookmarks if you do nothing, but beyond that users might want to get back their bookmarks after they accidentally deleted them all. To what extent should backup/versioning etc. feature in our roadmap?
  • • I hadn't considered versioning so this is just 'off the top of my head'. If we could do a "dated profile backup" every so often, plus before a Firefox upgrade, like a Windows Restore point, that would be pretty swell. - A
    • Options discussed: local snapshot of profile directory (because Sync doesn't sync per-machine stuff); snapshots to DropBox; snapshots to a Mozilla service. These are nice because it reduces the cost to the synchronization service.

• Is there a per-user uptime requirement? (Asa, mconnor)
  • mconnor: I would propose 99.9% for any given user's data access, at the high end. This is the high end of what is viable with a single datacenter, and multi-datacenter would be prohibitively expensive.
  • How would you define uptime requirement? A see many options. At any time N% of users have Sync working for them. Sync is working for any user N% of the time. Sync is available to a particular user N% of the time they attempt to connect to it. etc.
    • Can we separate out "uptime" from "times when the service needs to be up"? I mean, for example, if it's always up when I'm setting up a new device but it's somewhat less consistently up when I'm just making incremental updates? As we add "instant" items to this feature, like "send this tab to my phone now" we sure don't want to have a "sorry, couldn't connect, please wait for the service to return" message or anything like that. I'd also ask "what's industry standard" for things like this -- a high-profile consumer service. - A

• How should migration be messaged? Is it sufficient to have a note on the newly set-up device saying that your other devices need to be upgraded? A non-fatal notice on the old devices (requires work on Sync 1.1)? A fatal error on the old devices?
  • • That sounds pretty good. Could we somehow kick the updater in the other clients if we see that one client has moved forward a version, either triggering the update or prompting for a restart if the update's already staged? - A

• To what extent will we be pushing this feature? Privileged account + durable storage implies backup, and if half a billion users sign up, we need to reflect that in capacity planning! This is very relevant to mmayo's team, and affects the operational timeline.
  • • From a Product perspective, I want 100% of Firefox users using this service. I don't expect to reach that level of usage any time soon, but I'd say 10x what we currently have is the kind of boost I'm looking for in the short term. - A

• Do we want the PIdP to be featured as browser-wide UX, or only as part of Sync? (Asa, Karen)
  • This requires thought for technical delivery on mobile, too.
    • I'm not sure I understand this question. Tell me if this answer fits that question. We want users to create a Firefox account. If when they're creating that account, we see they're entering an existing Persona ID, we should prompt them to "upgrade" that ID (adding the Firefox Sync/Backup capability to that Persona ID) by re-typing their Persona password. Similarly, if at a later time the user encounters a site that utilizes Persona ID for logins, and the user starts typing in their Firefox ID, we can offer to "upgrade" their Firefox account to a Persona ID credential. On the backend they're the same thing just with one bit set for Firefox enabled and one bit set for Persona ID enabled. Thunder has this all sorted out -- assuming I'm answering the question you've asked. - A
    • Discussion result: PIdP will be a top-level entity; essentially a standalone product/project. A Firefox Account will be surfaced within, say, the Firefox menu. Sync functionality is essentially a checkbox: "oh, now you have a Firefox Account, which browser data do you want to be synchronized?". Need to discuss this with Fx leads.

Technical Questions

• Is the key storage (wrapped master key) service a part of the Preferred Idp API, or is it an Identity Attached Service? (Ben, mconnor)

• What is the timeline for delivering key-wrapping as a capability? (Ben, bwarner)

• Are any further changes required to the 2.0 protocol? (gps, rnewman, rfkelly)

• Are there any further changes required to the v6 storage format? (gps)

• How does the account/PIdP login work on the client -- is this a general feature for the browser, or a Sync-specific feature?
  • Answered: general feature.