CloudServices/Sync/FxSync/Developer/ClientAPI

Terms of Service

By accessing or using the Firefox Sync APIs in connection with the development of your own client software to access the Firefox Sync services (a “Third Party Client”), you acknowledge that you will need to install and use a local version of the Firefox Sync server for multiple account testing and that any use of Mozilla’s hosted Firefox Sync services is subject to Mozilla’s Firefox Sync Terms of Service at https://services.mozilla.com/tos/. Further, you agree (a) to maintain and link to (including on websites from which your Third Party Client may be downloaded) a separate, conspicuous, and reasonably detailed privacy policy detailing how data collected or transmitted by your Third Party Client is managed and protected; (b) that your Third Party Client will only store data in encrypted form on the Firefox Sync servers operated by Mozilla; (c) that you and your Third Party Client will use the Firefox Sync APIs solely for their intended purpose; (d) that your Third Party Client will not hide or mask its identity as it uses the Services and/or Firefox Sync APIs, including by failing to follow required identification conventions; and (e) that you and your Third Party Client will not use the Firefox Sync APIs for any application or service that replicates or attempts to replicate the Services or Firefox Sync experience unless such use is non-confusing (by non-confusing, we mean that people should always know with whom they are dealing and where the information or software they are downloading came from). You may not imply, either directly or by omission, that your Third Party Client is produced or endorsed by Mozilla. By providing access to the Firefox Sync APIs, Mozilla is not granting you a license to any of our trademarks.

Overview

This page describes how to the client-side Sync API for sync engines (and their helper classes). The focus is on using this API to create a new sync engine to synchronize a new data type. The data type can be anything that extension JS code has access to through any Mozilla API; this means this page must of necessity be pretty vague about reading and writing the underlying data. You'll have to fill in those blanks yourself. Try browsing the [link] xpcom documentation to find out how to get at the many types of useful data that Mozilla stores.

To sync a new data type, you'll need to write an engine class* that extends the base SyncEngine class; you'll also need to extend three helper classes. Here are the classes you need to extend, and the files in which they're defined:

1. SyncEngine, in services/sync/modules/engines.js
2. CryptoWrapper, in services/sync/modules/base_records/crypto.js
3. Store, in services/sync/modules/stores.js
4. Tracker, in services/sync/modules/trackers.js

It will be very helpful to look at the existing sync engines -- such as the one for bookmarks and the one for history -- and their helper classes, for guidance. You can find these files at:

• services/sync/modules/engines/bookmarks.js -- the BookmarkEngine, BookmarkStore, and BookmarkTracker.
• services/sync/modules/type_records/bookmark.js -- the Record classes for the various subtypes of bookmarks
• services/sync/modules/engines/history.js -- the HistoryEngine, HistoryStore, and HistoryTracker.
• services/sync/modules/type_records/history.js -- the HistoryRec record class.

After implementing your classes, you'll have to register them with Weave; you should also add a check-box to the Weave preferences screen to let the user turn your engine on and off. How to do these thing is explained at the end of the page.

• - Javascript is prototype-based, not class-based, so technically it's not correct to talk about "subclassing" or "extending a class", but what we're doing is very much the equivalent of that, and I don't know any other terminology to explain it as clearly.

Writing a Record class

A Record object is a wrapper around a single instance of whatever data it is that you are syncing -- a single bookmark in the case of the bookmark engine, a single browser tab in the case of the tab engine, and so on.

There's a class called CryptoWrapper defined in weave/modules/base_records/crypto.js, which handles all the encryption and decryption of your record for you. All you have to do is write a class that extends CryptoWrapper and maintains a property called cleartext. cleartext must be a JSON-dictionary-style object; put into it all values that you want to have encrypted, stored on the server, decrypted, and synced up.

You may find it useful to write getters and setters for various properties of your Record class.

The skeleton of a sample Record class:

 function FooRecord() {
   this._FooRecord_init();
 }
 FooRecord.prototype = {
   __proto__: CryptoWrapper.prototype,
   _logname: "Record.Foo", 

   _FooRecord_init: function FooItem_init(uri) {
     this._CryptoWrap_init(uri);
     this.cleartext = {};
   },

   get bar() this.cleartext.bar, 
   set bar(value) {
     this.cleartext.bar = value;
   }
 };

Writing a Store class

Your Store class (which extends Store, as defined in services/sync/modules/stores.js) has the job of creating and maintaining a set of Record objects from the underlying data. The store must also make updates to the underlying data itself, to keep the data up-to-date with the user's latest changes, when the store is instructed to do so.

The majority of the code you write for syncing a new data type will most likely be in the Store class.

Each Record that the Store keeps track of must be identified by a unique ID. This can be a true GUID (Globally Unique ID), but it doesn't have to be -- it only has to be unique among all records in the store. (So if an ID used for a bookmark record in the bookmark store is reused for a tab record in the tab store, that's fine.)

Nevertheless, most of the code refers to them as GUIDs, so I'll use GUID for the rest of this article.

Depending of what type of data you're working with, you might already have GUIDs built into your data that you can make use of. If not, you may have to invent your own mapping from data objects to GUIDs. In this case, you will probably find the makeGUID() function (in Utils) useful:

Cu.import("resource://services-sync/util.js");
// ...
let newGuid = Utils.makeGUID();

You can assign GUIDs to items in whatever way you see fit, as long as you are consistent with the mapping. It's entirely up to your Store class to define and maintain this mapping. Also keep in mind that although the data will be encrypted on the server, the GUIDs are public, so don't put passwords into them or anything.

Your Store class must implement the following methods:

• itemExists(id)
• createRecord(id, uri)
• changeItemId(oldId, newId)
• getAllIds()
• wipe()
• create(record)
• update(record)
• remove(record)

You may also find it useful to override other methods of the base class, depending on what you're doing.

Since this is a lot of methods to write, I'll break them down one by one in the following sub-sections.

createRecord

The createRecord( guid, uri ) method gets called by the engine to request a new record for an item with a given GUID. It's your Store's responsibility to instantiate a Record object (of the class you defined earlier), assign the given GUID, and return it.

itemExists(guid)

Should simply return true if an item with the given guid exists in the store, false otherwise.

changeItemId(oldId, newId)

Must find the stored item currently associated with the guid oldId and change it to be associated with the guid newId.

getAllIds()

Must return an iterable list containing the GUIDs of every item being stored on the local system. This can be a dictionary-type object where the keys are the GUIDs and the values are whatever; or it can simply be a list of GUIDs.

This is the method that the engine will call the first time it syncs, in order to get a complete inventory of what data there is that will need to be uploaded to the server. Unless the items you're syncing have their own inherent GUIDs already defined, you'll need to invent GUIDs for all your items at this time. When one of these GUIDs is later passed as an argument to createRecord(), you need to return a record based on the matching data. Therefore, it's the responsibility of this method to define the guid -> item mapping, and to store it for later reference by other methods.

wipe()

When this method is called, your Store needs to completely wipe all of its stored data from the local application. That doesn't mean just whatever caches of the data the Store happens to be maintaining; it means the underlying data itself. For instance, BookmarkStore.wipe() deletes all bookmarks from the current Firefox profile. How to do this for your particular data type is up to you to figure out.

Create / Update / Remove

The next three methods are called by the sync algorithm when it determines that the state of the local application needs to be changed to keep it up-to-date with the user's remote activities. The sync algorithm will call your Store object with a record to be created, updated, or removed, and it is your Store object's responsibility to apply this change to the local application using whatever methods are neccessary.

create(record)

Not to be confused with createRecord(), this method (which should probably be renamed for clarity soon) tells your Store to create a new item in the local application, based on the data in record. (So for example, BookmarkStore.create() adds a new bookmark to the Firefox profile).

update(record)

The argument is a record which has been remotely modified; your Store should locate the matching local item (presumably using the GUID, which is available in record.id) and update it to the new values provided in record.

remove(record)

The argument is a record which has been remotely deleted; your Store should locate the matching local item and delete it. (Don't rely on the record that is passed in to this method having any of its attributes set correctly except for .id. The .id should be al you need, anyway.)

Example Store class skeleton

 function FooStore() {
   // Maintains the store of all your Foo-type items and their GUIDs.
 }
 FooStore.prototype = {
   __proto__: Store.prototype,
   _logName: "foo",
   itemExists: function(guid) {
     // Return true if an item with given guid exists in the store.
   },
   createRecord: function(guid, uri) {
     record = new FooRecord(uri);
     // Look up the data corresponding to this guid, by the mapping
     // you've defined.
     // Set the data and the guid on the new record:
     record.bar = 17; // or whatever
     record.id = guid;
     // return the record
     return record;
   },
   changeItemId: function(oldId, newId) {
     // Find the item with guid = oldId and change its guid to newId.
   },
   getAllIds: function() {
     // Return a list of the GUIDs of all items.  Invent GUIDs for any items
     // that don't have them already, and remember the mapping for later use.
   },
   wipe: function() {
     // Delete everything!
   },
   create: function(record) {
     // Create a new item based on the values in record
   },
   update: function(record) {
     // look up the stored record with id = record.id, then set
     // its values to those of new record
   },
   remove: function(record) {
     // look up the stored record with id = record.id, then delete it.
   }
 };

Writing a Tracker class

Your tracker class must inherit from Tracker, which is defined in weave/engines/trackers.js. Its purpose in life is to track changes to whatever data type you are syncing. It must maintain a list of GUIDs for objects that have been changed and therefore require syncing. It also has to maintain a "score", which is a number from 0 to 100 which represents "how badly does this data type need to be synced right now".

Getting your tracker to track changes in the underlying data is probably easiest to do if you register your tracker as an observer, so that it can receive notifications from one or more Mozilla components. What events you listen for is entirely up to you. You can find out more about registering as an observer here: [link].

The Score

The score is stored in this._score, a variable defined by the base Tracker class. Set your score by assigning a value to this._score.

The score number is used by the scheduler to decide when your engine will be synced. Setting it to zero means "I have no need to sync". The higher you set this number, the more urgently the scheduler treats your request. Setting it to 100 means "Sync me immediately". It's up to you to figure out the best way to assign a score based on the current state of whatever data type you're tracking.

Your tracker score is automatically reset to zero after each time your engine syncs.

The changed GUID list

The base class maintains a list of GUIDs that need syncing. When your tracker detects that an item has changed, you should add it to this list by calling:

this.addChangedID(guid);

These GUIDs correspond to the .id fields of your Record objects; see the section on the Store class for more about defining and maintaining the mapping between GUIDs and Records.

Example

Here's the skeleton of a sample Tracker class.

 function FooTracker() {
   this._init();
 }
 FooTracker.prototype = {
   __proto__: Tracker.prototype,
   _logName: "FooTracker",
   file: "foo",

   _init: function FooTracker_init() {
     // The ugly syntax on the next line calls the base class's init method:
     this.__proto__.__proto__._init.call(this);
     /* Here is where you would register your tracker as an observer, so that
        its onEvent() (or other appropriately named) method can be called
        in response to events. */
   },

   onEvent: function FooTracker_onEvent() {
     /* Here is where you'd handle the event.  See the documentation for
        whatever service you are observing to find out what to call this
        method, what arguments to expect, and how to interpret them. */
     var guid = 0;

     /* Here is where you'd include code to figure out the GUID of the item
        that has changed... */
     this.addChangedID(guid);

     // Update the score as you see fit:
     this._score += 10;
   }
 };

Writing an Engine class

Your Engine class serves to tie together your Store, Tracker, and Record classes into a bundle that the Weave sync algorithm can instantiate and use.

You're probably sick of writing subclasses by this point, but don't worry: this one is very easy. I saved it for last because it requires the least code.

Your class must derive from the SyncEngine class, defined in weave/modules/engines.js. SyncEngine contains a lot of code which handles logic for the core sync algorithm, but your subclass won't need to call any of this directly, unless you are overriding part of the sync algorithm to provide custom sync behavior (an advanced technique outside the scope of this article).

A sample Engine class:

function FooEngine() {
  this._init(); 
}
FooEngine.prototype = {
  __proto__: SyncEngine.prototype,
  name: "foo",
  displayName: "Foo",
  logName: "Foo",
  _recordObj: FooRecord,
  _storeObj: FooStore,
  _trackerObj: FooTracker
};

As you can see, there isn't actually any new code here at all; the prototype simply defines some metadata such as the Store and Tracker classes to use, and the human-readable name that will be used in the log files to identify errors and status messages coming from this engine.

(Don't forget that you'll need to import SyncEngine and export FooEngine. So the top of your file should include:)

const EXPORTED_SYMBOLS = ["FooEngine", /* ... etc... */ ];
const Cu = Components.utils;
// etc...
Cu.import("resource://weave/engines.js");

Installing your classes into Weave

[TODO describe where to put all your code and how to make it into importable modules]

In the file weave/chrome/content/fx-weave-overlay.js, find the function FxWeaveGlue(), and within the try block import your Engine class, instantiate it, and register it with Weave. Like so:

 function FxWeaveGlue() {
   //... etc ...
   try {
     Cu.import("resource://weave/engines/bookmarks.js");
     Weave.Engines.register(new BookmarksEngine());

     Cu.import("resource://weave/engines/history.js");
     Weave.Engines.register(new HistoryEngine());

     //.. etc ...

     // Your engine here:
     Cu.import("resource://weave/engines/foo.js");
     Weave.Engines.register(new FooEngine());

   } catch (e) {
   //... etc ...
 // In the latest Weave, You can register your engine in anywhere that you  
 // can access the Weave.Engines.register function. Please be noted that the 
 // parameter you passed should be the name of your engine, such as 
 // Weave.Engines.register(FooEngine), but not new FooEngine().

To make your engine work in Fennec, you must also add the same code to the function FennecWeaveGlue() in the file weave/chrome/content/fennec-weave-overlay.js.

Updating the Weave preferences screen

Add a preference to turn your sync engine on/off by adding a line to weave/defaults/preferences/sync.js like so:

pref("extensions.weave.engine.foo", true);

Default to true or false as you see fit.

Add a check box to the preferences screen to enable the user to change this setting, by adding a line to weave/chrome/content/fx-preferences.xul like this:

<preference id="extensions.weave.engine.passwords"
            name="extensions.weave.engine.passwords"
            type="bool"/>

right after all the other similar preferences.

Finally, the preferences screen for Fennec is constructed separately, so to add your check box to the Fennec prefs screen as well you'll need to edit one more file. Open up weave/chrome/content/fennec-preferences.xul and add:

 <richpref pref="extensions.weave.engine.foo"
	   title="Foo" type="bool">
   Sync foo
 </richpref>

right after the similar preferences.

[TODO: Explain better where to add these.]

Testing and Debugging your Engine

TODO