Services/Sync/Developer/StorageFormat

Under a user's /storage path, there's a set of well-known records and storage formats used by the Weave Sync add-on. This document describes those records and their formats, e.g., JS-object format of the JSON-string payload.

Weave Basic Object

All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the Sync Server API, but for a given client storage version, only a subset may be used.

Version 2

The following describes the JS-object represented by the JSON-string record:

id	string	Identifying string for the record
modified	number	Time when this record was last changed; set by the server
sortindex	number	Relative importance of this record
payload	string	String with data; usually a JSON-string

Changes from v1 -> v2

The client no longer stores and accesses the parentid and predecessorid fields in the WBO and instead stores them encrypted inside the Browser Object. Only bookmarks used these fields.

Version 1

The following describes the JS-object represented by the JSON-string record:

id	string	Identifying string for the record
parentid	string	Record id of a "parent"
predecessorid	string	Record id of a "predecessor"
modified	number	Time when this record was last changed; set by the server
sortindex	number	Relative importance of this record
payload	string	String with data; usually a JSON-string

Payload: Encrypted Data Object

Individual data engines, e.g., bookmarks, encrypt their Browser Objects payloads before packing it into the payload field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.

Version 2

The following describes the JS-object represented by the JSON-string payload:

ciphertext	string	Encrypted JSON-stringified Browser Object
encryption	string	Url of the crypto record containing the symmetric key to decrypt the ciphertext
IV	string	Initialization vector used when decrypting the ciphertext
hmac	string	SHA256 HMAC of the ciphertext and key derived from the decrypted base64 symmetric key string that would be used to decrypt the ciphertext

Once the ciphertext is decrypted to get the Browser Object data, there are additional fields in the JS-object of the decrypted JSON-string:

id	string	The original id used when encrypting the payload to ensure the encrypted data is of the requested record
deleted	boolean	Set to true if this record represents a "delete" action

All client Browser Object data like bookmarks, history, and now additionally "clients" are encrypted.

Changes from v1 -> v2

The decrypted ciphertext JSON-string no longer wraps the Browser Object data with an extra [array].

At the payload level, there are two additional fields: IV and hmac. The IV is stored per-record instead of on the /crypto/<engine> so that a different IV can be used per record. The hmac can be used to verify that the encrypted payload has not been tampered with.

Once the ciphertext inside the payload is decrypted to access the Browser Object's payload, there are fields in addition to the usual Browser Object fields for that data type: id and deleted. For all encrypted data objects, there is an id field, which can be used to verify that the encrypted data is for the requested object. Instead of treating empty string payloads as "delete" records, the deleted field will be set to true.

The clients data is no longer a cleartext JSON payload and instead encrypts its Browser Object data like any other encrypted-data engine.

Version 1

The following describes the JS-object represented by the JSON-string payload:

ciphertext	string	Encrypted JSON-stringified Browser Object, which has been first wrapped with an extra [array], i.e., encrypt(stringify([browserObject]))
encryption	string	Url of the crypto record containing the symmetric key to decrypt the ciphertext

Empty string "" payloads represent a "delete" action for the record id.

Payload: meta/global

The payload of the meta/global record contains general metadata to describe data like versions and syncID.

Version 2

The following describes the JS-object represented by the JSON-string payload:

storageVersion	number
syncID	string
engines	object

Changes from v1 -> v2

The storageVersion is now an integer (2) instead of a string, e.g., "1.1", and will only change when the storage format changes. It will no longer automatically changed to the newest client to write the record and instead only change on an incompatible storage format change.

There is an engines object that has engine names as its fields with corresponding value objects with two fields: syncID and version. The syncID for an engine changes when the data for that engine drastically changes, and the client will respond by re-syncing all of its data. The version of an engine acts like the storageVersion and indicates the Browser Object version for the corresponding data.

Version 1

The following describes the JS-object represented by the JSON-string payload:

storageVersion	string
syncID	string

Payload: keys/pubkey

This is the public key part of the keypair used to encrypt the symmetric keys used to encrypt data.

Version 2

The following describes the JS-object represented by the JSON-string payload:

keyData	string
privateKeyUri	string
type	string

Changes from v1 -> v2

No changes.

Version 1

The following describes the JS-object represented by the JSON-string payload:

keyData	string
privateKeyUri	string
type	string

Payload: keys/privkey

This is the private key part of the keypair used to decrypt the symmetric keys used to decrypt data. The private key is encrypted with a key derived from the secret phrase.

Version 2

The following describes the JS-object represented by the JSON-string payload:

salt	string
iv	string
keyData	string
privateKeyUri	string
type	string

Changes from v1 -> v2

No changes.

Version 1

The following describes the JS-object represented by the JSON-string payload:

salt	string
iv	string
keyData	string
privateKeyUri	string
type	string

Payload: crypto/<engine>

This keyring contains the symmetric key used to encrypt/decrypt Encrypted Data Objects.

Version 2

The following describes the JS-object represented by the JSON-string payload:

keyring	object

Changes from v1 -> v2

There is only one field, keyring, in the payload now that bulkIV is removed.

The keyring object still has field entries of the key to decrypt the wrapped symmetric key, but instead of the wrapped key being the value of the field, it is now an object with two fields: wrapped and hmac. The wrapped symmetric key from v1 is now the value of wrapped. The hmac can be used to verify that the wrapped key has not been tampered with.

Version 1

The following describes the JS-object represented by the JSON-string payload:

bulkIV	string
keyring	object