Services/Sync/Developer/StorageFormat

Under a user's /storage path, there's a set of well-known records and storage formats used by the Weave Sync add-on. This document describes those records and their formats, e.g., JS-object format of the JSON-string payload.

The storage version of the data is stored as part of the meta/global payload.

Weave Basic Object

All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the Sync Server API, but for a given client storage version, only a subset may be used.

Versions 2, 3, and 5

The following describes the JS-object represented by the JSON-string record:

id	string	Record identifier. Starting with version 5 this SHOULD be exactly 12 characters from the base64url alphabet.
modified	number	Time when this record was last changed; set by the server
sortindex	number	Relative importance of this record
payload	string	String with data; usually a JSON-string
ttl	integer (optional)	The number of seconds to keep this record. After that time, this item will not be returned.

Example

{"id":"{3ab32a23-822d-424c-a4b8-88da8cf93eb2}0",
 "modified":1278109839.96,
 "sortindex":140,
 "payload":"{\"ciphertext\":\"e2zLWJYX\/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN\/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o\/ax\/3B2cM+6J2Frj6Q83OEW\/QBC8Q6\/XHgtJJlFi6fKWrG+XtFxS2\/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=\",\"encryption\":\"..\/crypto\/bookmarks\",\"IV\":\"GluQHjEH65G0gPk\/d\/OGmg==\",\"hmac\":\"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289\"}"}

Changes from v1 -> v2

The client no longer stores and accesses the parentid and predecessorid fields in the WBO and instead stores them encrypted inside the Browser Object. Only bookmarks used these fields.

Version 1

The following describes the JS-object represented by the JSON-string record:

id	string	Opaque identifying string for the record
parentid	string	Record id of a "parent"
predecessorid	string	Record id of a "predecessor"
modified	number	Time when this record was last changed; set by the server
sortindex	number	Relative importance of this record
payload	string	String with data; usually a JSON-string

---

Payload: Encrypted Data Object

Individual data engines, e.g., bookmarks, encrypt their Browser Objects payloads before packing it into the payload field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.

Version 5

Version 5 is similar to Version 2 and 3, except:

• the encryption field (which specifies the key to use) has been dropped; key selection is now implied by the WBO's collection
• the HMAC is now calculated with a separate key, used in its raw byte form rather than in base64 encoding.

The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle is used.

ciphertext	string	Encrypted JSON-stringified Browser Object
IV	string	Initialization vector used when decrypting the ciphertext
hmac	string	SHA256 HMAC in hex representation, computed on the base64 encoded version of the ciphertext, using the byte-representation of the HMAC key from the same bundle as the encryption key.

Example

{"ciphertext":"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=",
 "IV":"GluQHjEH65G0gPk/d/OGmg==",
 "hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}

Version 3

Like Version 2, except encryption is now a relative URL (relative to the Weave Basic Object's URL).

Example

{"ciphertext":"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=",
 "encryption":"../crypto/bookmarks",
 "IV":"GluQHjEH65G0gPk/d/OGmg==",
 "hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}

Version 2

The following describes the JS-object represented by the JSON-string payload:

ciphertext	string	Encrypted JSON-stringified Browser Object
encryption	string	Url of the crypto record containing the symmetric key to decrypt the ciphertext
IV	string	Initialization vector used when decrypting the ciphertext
hmac	string	SHA256 HMAC in hex format, computed on the base64 encoded version of the ciphertext, using a base64 encoded version of the key used to encrypt the ciphertext in the first place as the key to the hmac algorithm.

Once the ciphertext is decrypted to get the Browser Object data, there are additional fields in the JS-object of the decrypted JSON-string:

id	string	The original id used when encrypting the payload to ensure the encrypted data is of the requested record
deleted	boolean	Set to true if this record represents a "delete" action

All client Browser Object data like bookmarks, history, and now additionally "clients" are encrypted.

Changes from v1 -> v2

The decrypted ciphertext JSON-string no longer wraps the Browser Object data with an extra [array].

At the payload level, there are two additional fields: IV and hmac. The IV is stored per-record instead of on the /crypto/<engine> so that a different IV can be used per record. The hmac can be used to verify that the encrypted payload has not been tampered with.

Once the ciphertext inside the payload is decrypted to access the Browser Object's payload, there are fields in addition to the usual Browser Object fields for that data type: id and deleted. For all encrypted data objects, there is an id field, which can be used to verify that the encrypted data is for the requested object. Instead of treating empty string payloads as "delete" records, the deleted field will be set to true.

The clients data is no longer a cleartext JSON payload and instead encrypts its Browser Object data like any other encrypted-data engine.

Version 1

The following describes the JS-object represented by the JSON-string payload:

ciphertext	string	Encrypted JSON-stringified Browser Object, which has been first wrapped with an extra [array], i.e., encrypt(stringify([browserObject]))
encryption	string	Url of the crypto record containing the symmetric key to decrypt the ciphertext

Empty string "" payloads represent a "delete" action for the record id.

---

Payload: meta/global

The payload of the meta/global record contains general metadata to describe data like versions and syncID.

Version 2, 3 and 5

The following describes the JS-object represented by the JSON-string payload:

storageVersion	number	Integer version that can stay the same across multiple client versions so that different versioned clients can sync with each other
syncID	string	Opaque string that changes when drastic changes happen to the overall data; this causes the client to drop cached keys/data
engines	object	A hash with fields of engine names and values of objects that contain version and syncID and behave like the version/syncID of this payload but on a per-engine level

Example

{"syncID":"JnvqPEn(6~",
 "storageVersion":3,
 "engines":{"clients":{"version":1,"syncID":"LwjtCQjdsV"},
            "bookmarks":{"version":1,"syncID":"ApPN6v8VY4"},
            "forms":{"version":1,"syncID":"UKeuhB.aOZ"},
            "tabs":{"version":1,"syncID":"G!nU*7H.7j"},
            "history":{"version":1,"syncID":"9Tvy_Vlb44"},
            "passwords":{"version":1,"syncID":"yfBi2v7Pp)"},
            "prefs":{"version":1,"syncID":"*eONx!6GXA"}}}

Changes from v1 -> v2

The storageVersion is now an integer (2) instead of a string, e.g., "1.1", and will only change when the storage format changes. It will no longer automatically changed to the newest client to write the record and instead only change on an incompatible storage format change.

There is an engines object that has engine names as its fields with corresponding value objects with two fields: syncID and version. The syncID for an engine changes when the data for that engine drastically changes, and the client will respond by re-syncing all of its data. The version of an engine acts like the storageVersion and indicates the Browser Object version for the corresponding data.

Version 1

The following describes the JS-object represented by the JSON-string payload:

storageVersion	string	Version of the newest client that synced
syncID	string	Opaque string that changes when drastic changes happen to the overall data; this causes the client to drop cached keys/data

---

Payload: crypto/keys

In storage Version 5, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate keys and kept in key pairs.

The keys WBO is encrypted and verified just like any other WBO, except a different key bundle is used. The key bundle for the keys WBO is derived from the Sync Key using an HKDF with HMAC-SHA256 as the HMAC function (see RFC 5869):

Pseudo-code:

 HMAC_INPUT = "Sync-AES_256_CBC-HMAC256"
 encryption_key = HMAC-SHA256(sync_key, "" + HMAC_INPUT + username + "\x01")
 hmac_key = HMAC-SHA256(sync_key, encryption_key + HMAC_INPUT + username + "\x02")

Here sync_key is the 16 byte representation of the Sync Key. To translate between the byte and user-readable translation, base32 is used, although with a slightly different alphabet than what RFC 4648 uses. For readability reasons, 'l' has been replaced with '8' and 'o' with '9':

 sync_key = decodeBase32(sync_key_ui.replace('8', 'l').replace('9', 'o'))
 sync_key_ui = encodeBase32(sync_key).replace('l', '8').replace('o', '8)

Version 5

default	array	Default key pair: [encryption key, HMAC key]
collections	object	Mapping of collection name to collection-specific key pairs: [encryption key, HMAC key]
collection	string	Currently defaulting to "crypto"

Example

{"id":"keys",
 "collection":"crypto",
 "collections":{},
 "default:['dGhlc2UtYXJlLWV4YWN0bHktMzItY2hhcmFjdGVycy4=',
           'eWV0LWFub3RoZXItc2V0LW9mLTMyLWNoYXJhY3RlcnM=']}

Payload: keys/pubkey

This is the public key part of the keypair used to encrypt the symmetric keys used to encrypt data.

Version 3

Like Version 2, except that privateKeyUri is now relative to the public key's URL.

Example

{"type":"pubkey",
 "keyData":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArd0DVRP1Ve1ZLZDbdcJvuf/S/zdsy+9qwakS0A2z6D2aUEd0uH61SacUw4StxibSYDcU+4M+zsCZugONoSUCBGcxKAhqStcpxY8SPTwAc3CTMFiTVMHL7rtjq9NYhYT2qFBdrLOmS3nCczJYjTzdig6RhA+GrHzaqmXPXJakQ8uyVYpsWfKj++o+VzNKLXlxRhmw170hzXgtuwVBK5iHCj+ndEQvVOlTQ1g7YFGiMMtOYs4fpc6CFcZm3w4FdmjADluzPk+ZugQkn/Sz0/JZInb0DnSHuRAt86PLfnDiMHQ5GwJynGsE/EYCYZrWrsTp/6y7OBx1Qk4tBuQhbTHtewIDAQAB",
 "privateKeyUri":"privkey"}

Version 1 and 2

The following describes the JS-object represented by the JSON-string payload:

keyData	string	Value of the public key
privateKeyUri	string	Url to the corresponding private key
type	string	Type of the key: "pubkey"

---

Payload: keys/privkey

This is the private key part of the keypair used to decrypt the symmetric keys used to decrypt data. The private key is encrypted with a key derived from the secret phrase.

Version 3

Like Version 2, except publicKeyUri is now relative to the private key's URL.

Example

{"type":"privkey",
 "salt":"Ag76sNXjS4TVj9r2CeEo/w==",
 "iv":"OX3JxwaP1U5Hh+0qaHRtfg==",
 "keyData":"zOAhty69Qvjl49jjCntvlYK88Hv+KBgrfx6o9HIXdxplSkwOSS6Q2P1ee6KXZor7lBEzuedNMJbog3VrKkNwvTKe5DVuNx8UJVujn1C/A6FR6ml3v/Vimg2tC3MfUHdCBQcN1MX95RM2comwaUdPz+5iR3VtoqE8wHRV+yB7kXD4pbewepQaqCqKrzsZDSGGrBD0LyhQF/YuDkcwadRJs0t2JCEOMjQtqys7nbjOmAOuC4duL/1iWjb6FaTUAfB2CUI0okpjNkLf7XHHiJZOwjYEkpmoGCEXZZvwXtdj3wu30v8EL2tfdGWh7eYhy715fbcMWGBsF6gDpUjwjF+ihfezYZe9FcmA/8neSNJgzoXO4NwNMNBUKCJcWt2y8gZUhLzTvVf103oKqu/bzHQTihcU6RMLU1HVRbr6H3qeck26pDcSqk/ZRJld0WFtx9A+pVD1gzEwCLQ4BCmVMoMMFdyXVH4k0boxjckvIZUUeLpQeGVUj4xb86n48pTZco5AETmRqqVfGK31WjFY9m83+QgwqMCZR9qFMMVJPeyE8tpoqQXhxfJrNyHITgye/EPtVS5s0iLfWq/Qe9jHvIfKa5vlXm5kxyMBHacD07QrZqh4sy8vGPmJfwY8BmhcbeGjqx9yANX5O0KjVJlk2NtJ1yFd4PcMd4VJuVB3hMUPQwP6eZSfe+gj2n2epnL+KesaL+OgD8YRFg9ZyBPqz6VaHfSdLw5e/vmhLzLyiZkYT9K5/7UqFm/7W/VpMselZV7O1sYojIbWeIOcHtJ9wqeOQ20nl6phQJVyKLOY7ApOsLoqYfzkZoG/CrgYKheY20niiwqH+qG6V/rMGaM4RCfpI/2bmWJkhlxyuv+20yMWzRad3H1WpfgxMF0jz4OA3QQKxB+dF/+/50mpveJs7Plgfm3zH/whl+gfOAX6JqGJBFfCA7XUwxDW4EMyGnYj6YMbv3iTHxA30KpA//KmpdF+x6qCP3kbrBHLpNcCk8fqqP6xwojophid/lTieOyEgNL8iGTRpF+XICi+CJNFrIpGdONrmXcGMql2c28V3azX1NGscmXUZstNp0ckw9IuxJovgATgiOTUM+QLDjq2QVCP3Jxol+klK+0mHUGqVCMeQaEb/aG71LbPQNLNBk8SzT59QbBiCOgc6zVx0TtYoLNCbL0hMrocGAHZw1W82y24uAPuDJKMfyVEWJ4HBn6sonzeag1M2WuGJVJJ+o+QuxE1wXaPIYfyzWc6T3741TIc7gl669YjMvdbGxtC57/bjX+heWJyedxlbL9mtoPEApfWsYxrVRGodGXgeYf365HGhU6wjRU4Nq3Tkg/ptEbfPYDv1E83cEwc9UE7yGFXucQZbfggrxrLttPMNABX6ZWnsNOIiuQcolxcebjpBjDQJoE9GD+4XKFXCLcd7gMaS9EOgCig5o9sk6uC2D3jJWVlgCiZoFns2tqW8x2EvJFCPor4hsAgNqMEiyyLwxEiV8EYEnx1uFxxjF4irTY/TzKF0msPNpBuplra26qefrAcqXYgXH2bdx1jnfH7DRMijz41XcDcqWi+VR14SuSdoqT4WYHBJE8f0U9Ng8TPSvk9QLed8eFJmpirDRYTvXkJ9DJP9E39wvAH/W+6cVM//qCENyU=",
 "publicKeyUri":"pubkey"}

Version 1 and 2

The following describes the JS-object represented by the JSON-string payload:

salt	string	Salt used with the secret phrase
iv	string	Initialization vector used when decrypting the keyData
keyData	string	Encrypted private key that needs the secret phrase, salt, and iv to decrypt
publicKeyUri	string	Url to the corresponding public key
type	string	Type of the key: "privkey"

---

Payload: crypto/<engine>

This keyring contains the symmetric key used to encrypt/decrypt Encrypted Data Objects.

Version 3

Like Version 2, except the URLs in the keyring hash are now relative to the symmetric key's URL.

Example

{"keyring":{"../keys/pubkey":{"wrapped":"P3nQWhQtUHpUpdzI8RD0DkW3OSnNcrcOGhWBM6p7a9imIJ3K8RpsEVnhCIBmkprg40lddRgH/o+vDi2jYovxkFKUzM3izLDa016JZud7GWlx50WNpxaWmst76DlXtjnLDK44tJw2UzRJI6jKq/k5rm8anUNrBYWqS7v97+OiHG4viFI9IQCvRrCY0ow+Z37NppoArW/kbna8Dl4UQAkVlQNOgTjvWW6BtiXd3HN7iA4LFgGTJCuQhfMAZel0fRq9vkW6XCZr6OJqGeEXbfmkfwAMQ3S0THCeki6ejAhiGW70OXF+gBqBrYvTiqyVXuRNKRY6TtH+9g8Vb9aj2O6F/w==",
                              "hmac":"682bac23003ceaf660391b8342bb423f2e52200b238eacb9640dc39e69c72a72"}}}

Version 2

The following describes the JS-object represented by the JSON-string payload:

keyring	object	A hash with fields of key urls used to decrypt the field's value of an object containing the wrapped symmetric key

The following describes the JS-object for keyring:

wrapped	string	The encrypted symmetric key that is decrypted using the key located at the url that indexed to this entry
hmac	string	SHA256 HMAC of wrapped and the key derived from the decrypted base64 private key string that would be used to decrypt the wrapped key

Changes from v1 -> v2

There is only one field, keyring, in the payload now that bulkIV is removed.

The keyring object still has field entries of the key to decrypt the wrapped symmetric key, but instead of the wrapped key being the value of the field, it is now an object with two fields: wrapped and hmac. The wrapped symmetric key from v1 is now the value of wrapped. The hmac can be used to verify that the wrapped key has not been tampered with.

Version 1

The following describes the JS-object represented by the JSON-string payload:

bulkIV	string	Initialization vector used to encrypt/decrypt all data for the corresponding engine
keyring	object	A hash with fields of key urls used to decrypt the field's value of a wrapped symmetric key