Services/Sync/Developer/StorageFormat

Under a user's /storage path, there's a set of well-known records and storage formats used by the Weave Sync add-on. This document describes those records and their formats, e.g., JS-object format of the JSON-string payload.

Weave Basic Object

All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the Sync Server API, but for a given client storage version, only a subset may be used.

Version 2

id	string
modified	float
sortindex	integer
payload	string

Changes from v1 -> v2

The client no longer stores and accesses the parentid and predecessorid fields in the WBO and instead stores them encrypted inside the Browser Object. Only bookmarks used these fields.

Payload: Encrypted Data Object

Individual data engines, e.g., bookmarks, encrypt their Browser Objects payloads before packing it into the payload field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.

Changes from v1 -> v2

The decrypted ciphertext JSON-string no longer wraps the Browser Object data with an extra [array].

At the payload level, there are two additional fields: IV and hmac. The IV is stored per-record instead of on the /crypto/<engine> so that a different IV can be used per record. The hmac can be used to verify that the encrypted payload has not been tampered with.

Once the ciphertext inside the payload is decrypted to access the Browser Object's payload, there are fields in addition to the usual Browser Object fields for that data type: id and deleted. For all encrypted data objects, there is an id field, which can be used to verify that the encrypted data is for the requested object. Instead of treating empty string payloads as "delete" records, the deleted field will be set to true.

The clients data is no longer a cleartext JSON payload and instead encrypts its Browser Object data like any other encrypted-data engine.

Payload: meta/global

The payload of the meta/global record contains general metadata to describe data like versions and syncID.

Version 2

storageVersion	integer
syncID	string
engines	object

Changes from v1 -> v2

The storageVersion is now an integer (2) instead of a string, e.g., "1.1", and will only change when the storage format changes. It will no longer automatically changed to the newest client to write the record and instead only change on an incompatible storage format change.

There is an engines object that has engine names as its fields with corresponding value objects with two fields: syncID and version. The syncID for an engine changes when the data for that engine drastically changes, and the client will respond by re-syncing all of its data. The version of an engine acts like the storageVersion and indicates the Browser Object version for the corresponding data.

Version 1

storageVersion	string
syncID	string

Payload: keys/pubkey

Changes from v1 -> v2

No changes.

Payload: keys/privkey

Changes from v1 -> v2

No changes.

Payload: crypto/<engine>

Changes from v1 -> v2

There is only one field, keyring, in the payload now that bulkIV is removed.

The keyring object still has field entries of the key to decrypt the wrapped symmetric key, but instead of the wrapped key being the value of the field, it is now an object with two fields: wrapped and hmac. The wrapped symmetric key from v1 is now the value of wrapped. The hmac can be used to verify that the wrapped key has not been tampered with.