MozillaWiki

Services/Sync/Developer/StorageFormat: Difference between revisions

Page Discussion

Services/Sync/Developer/StorageFormat (view source)

Revision as of 00:40, 11 August 2011

7,525 bytes added ,  11 August 2011
m
→‎Version 5
 
(21 intermediate revisions by 2 users not shown)
Line 7: Line 7:
All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the [[Labs/Weave/API|Sync Server API]], but for a given client storage version, only a subset may be used.
All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the [[Labs/Weave/API|Sync Server API]], but for a given client storage version, only a subset may be used.


== Version 2 and 3 ==
== Versions 2, 3, and 5  ==


The following describes the JS-object represented by the JSON-string record:
The following describes the JS-object represented by the JSON-string record:  


{| cellpadding=5
{| cellpadding="5"
! id
|-
| string
! id  
| Opaque identifying string for the record
| string  
| Record identifier. Starting with version 5 this SHOULD be exactly 12 characters from the base64url alphabet.
|-
|-
! modified
! modified  
| number
| number  
| Time when this record was last changed; set by the server
| Time when this record was last changed; set by the server
|-
|-
! sortindex
! sortindex  
| number
| number  
| Relative importance of this record
| Relative importance of this record
|-
|-
! payload
! payload  
| string
| string  
| String with data; usually a JSON-string
| String with data; usually a JSON-string
|-
! ttl
| integer (optional)
| The number of seconds to keep this record. After that time, this item will not be returned.
|}
|}


=== Changes from v1 -> v2 ===
=== Example  ===
 
<pre>
{"id":"{3ab32a23-822d-424c-a4b8-88da8cf93eb2}0",
"modified":1278109839.96,
"sortindex":140,
"payload":"{\"ciphertext\":\"e2zLWJYX\/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN\/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o\/ax\/3B2cM+6J2Frj6Q83OEW\/QBC8Q6\/XHgtJJlFi6fKWrG+XtFxS2\/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=\",\"encryption\":\"..\/crypto\/bookmarks\",\"IV\":\"GluQHjEH65G0gPk\/d\/OGmg==\",\"hmac\":\"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289\"}"}</pre>
 
=== Changes from v1 -&gt; v2 ===


The client no longer stores and accesses the <tt>parentid</tt> and <tt>predecessorid</tt> fields in the WBO and instead stores them encrypted inside the Browser Object. Only bookmarks used these fields.
The client no longer stores and accesses the <tt>parentid</tt> and <tt>predecessorid</tt> fields in the WBO and instead stores them encrypted inside the Browser Object. Only bookmarks used these fields.
Line 70: Line 83:


Individual data engines, e.g., bookmarks, encrypt their [[Labs/Weave/Developer/BrowserObjects|Browser Objects]] payloads before packing it into the <tt>payload</tt> field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.
Individual data engines, e.g., bookmarks, encrypt their [[Labs/Weave/Developer/BrowserObjects|Browser Objects]] payloads before packing it into the <tt>payload</tt> field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.
== Version 5 ==
Version 5 is similar to Version 2 and 3, except:
* the '''encryption''' field (which specifies the key to use) has been dropped; key selection is now implied by the WBO's collection
* the HMAC is now calculated with a separate key, used in its raw byte form rather than in base64 encoding.
The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle is used.
{| cellpadding=5
! ciphertext
| string
| Encrypted JSON-stringified Browser Object
|-
! IV
| string
| Initialization vector used when decrypting the <tt>ciphertext</tt>
|-
! hmac
| string
| SHA256 HMAC in hex representation, computed on the base64 encoded version of the ciphertext, using the byte-representation of the HMAC key from the same bundle as the encryption key.
|}
=== Example ===
<pre>{"ciphertext":"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=",
"IV":"GluQHjEH65G0gPk/d/OGmg==",
"hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}</pre>


== Version 3 ==
== Version 3 ==


Like Version 2, except '''encryption''' is now a relative URL (relative to the Weave Basic Object's URL).
Like Version 2, except '''encryption''' is now a relative URL (relative to the Weave Basic Object's URL).
=== Example ===
<pre>{"ciphertext":"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=",
"encryption":"../crypto/bookmarks",
"IV":"GluQHjEH65G0gPk/d/OGmg==",
"hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}</pre>


== Version 2 ==
== Version 2 ==
Line 94: Line 144:
! hmac
! hmac
| string
| string
| SHA256 HMAC computed on the base64 encoded version of the ciphertext, using a base64 encoded version of the key used to encrypt the ciphertext in the first place as the key to the hmac algorithm.  In addition, the hmac is stored in the weave object in HEX format, not base64 like everything else.
| SHA256 HMAC in hex format, computed on the base64 encoded version of the ciphertext, using a base64 encoded version of the key used to encrypt the ciphertext in the first place as the key to the hmac algorithm.
|}
|}


Line 145: Line 195:
The payload of the <tt>meta/global</tt> record contains general metadata to describe data like versions and syncID.
The payload of the <tt>meta/global</tt> record contains general metadata to describe data like versions and syncID.


== Version 2 and 3 ==
== Version 2, 3 and 5 ==


The following describes the JS-object represented by the JSON-string payload:
The following describes the JS-object represented by the JSON-string payload:
Line 162: Line 212:
| A hash with fields of engine names and values of objects that contain <tt>version</tt> and <tt>syncID</tt> and behave like the version/syncID of this payload but on a per-engine level
| A hash with fields of engine names and values of objects that contain <tt>version</tt> and <tt>syncID</tt> and behave like the version/syncID of this payload but on a per-engine level
|}
|}
=== Example ===
<pre>{"syncID":"JnvqPEn(6~",
"storageVersion":3,
"engines":{"clients":{"version":1,"syncID":"LwjtCQjdsV"},
            "bookmarks":{"version":1,"syncID":"ApPN6v8VY4"},
            "forms":{"version":1,"syncID":"UKeuhB.aOZ"},
            "tabs":{"version":1,"syncID":"G!nU*7H.7j"},
            "history":{"version":1,"syncID":"9Tvy_Vlb44"},
            "passwords":{"version":1,"syncID":"yfBi2v7Pp)"},
            "prefs":{"version":1,"syncID":"*eONx!6GXA"}}}</pre>


=== Changes from v1 -> v2 ===
=== Changes from v1 -> v2 ===
Line 186: Line 248:


----
----
= Payload: crypto/keys =
In storage Version 5, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate keys and kept in key pairs.
The '''keys''' WBO is encrypted and verified just like any other WBO, except a different key bundle is used. The key bundle for the '''keys''' WBO is derived from the Sync Key using an HKDF with HMAC-SHA256 as the HMAC function (see [http://tools.ietf.org/html/rfc5869 RFC 5869]):
Pseudo-code:
  HMAC_INPUT = "Sync-AES_256_CBC-HMAC256"
  encryption_key = HMAC-SHA256(sync_key, "" + HMAC_INPUT + username + "\x01")
  hmac_key = HMAC-SHA256(sync_key, encryption_key + HMAC_INPUT + username + "\x02")
Here '''sync_key''' is the 16 byte representation of the Sync Key. To translate between the byte and user-readable translation, base32 is used, although with a slightly different alphabet than what [http://tools.ietf.org/html/rfc4648 RFC 4648] uses. For readability reasons, 'l' has been replaced with '8' and 'o' with '9':
  sync_key = decodeBase32(sync_key_ui.replace('8', 'l').replace('9', 'o'))
  sync_key_ui = encodeBase32(sync_key).replace('l', '8').replace('o', '8)
== Version 5 ==
{| cellpadding=5
! default
| array
| Default key pair: [encryption key, HMAC key]
|-
! collections
| object
| Mapping of collection name to collection-specific key pairs: [encryption key, HMAC key]
|-
! collection
| string
| Currently defaulting to "crypto"
|}
=== Example ===
{"id":"keys",
  "collection":"crypto",
  "collections":{},
  "default:['dGhlc2UtYXJlLWV4YWN0bHktMzItY2hhcmFjdGVycy4=',
            'eWV0LWFub3RoZXItc2V0LW9mLTMyLWNoYXJhY3RlcnM=']}


= Payload: keys/pubkey =
= Payload: keys/pubkey =
Line 194: Line 298:


Like Version 2, except that '''privateKeyUri''' is now relative to the public key's URL.
Like Version 2, except that '''privateKeyUri''' is now relative to the public key's URL.
=== Example ===
<pre>{"type":"pubkey",
"keyData":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArd0DVRP1Ve1ZLZDbdcJvuf/S/zdsy+9qwakS0A2z6D2aUEd0uH61SacUw4StxibSYDcU+4M+zsCZugONoSUCBGcxKAhqStcpxY8SPTwAc3CTMFiTVMHL7rtjq9NYhYT2qFBdrLOmS3nCczJYjTzdig6RhA+GrHzaqmXPXJakQ8uyVYpsWfKj++o+VzNKLXlxRhmw170hzXgtuwVBK5iHCj+ndEQvVOlTQ1g7YFGiMMtOYs4fpc6CFcZm3w4FdmjADluzPk+ZugQkn/Sz0/JZInb0DnSHuRAt86PLfnDiMHQ5GwJynGsE/EYCYZrWrsTp/6y7OBx1Qk4tBuQhbTHtewIDAQAB",
"privateKeyUri":"privkey"}</pre>


== Version 1 and 2 ==
== Version 1 and 2 ==
Line 224: Line 334:


Like Version 2, except '''publicKeyUri''' is now relative to the private key's URL.
Like Version 2, except '''publicKeyUri''' is now relative to the private key's URL.
=== Example ===
<pre>{"type":"privkey",
"salt":"Ag76sNXjS4TVj9r2CeEo/w==",
"iv":"OX3JxwaP1U5Hh+0qaHRtfg==",
"keyData":"zOAhty69Qvjl49jjCntvlYK88Hv+KBgrfx6o9HIXdxplSkwOSS6Q2P1ee6KXZor7lBEzuedNMJbog3VrKkNwvTKe5DVuNx8UJVujn1C/A6FR6ml3v/Vimg2tC3MfUHdCBQcN1MX95RM2comwaUdPz+5iR3VtoqE8wHRV+yB7kXD4pbewepQaqCqKrzsZDSGGrBD0LyhQF/YuDkcwadRJs0t2JCEOMjQtqys7nbjOmAOuC4duL/1iWjb6FaTUAfB2CUI0okpjNkLf7XHHiJZOwjYEkpmoGCEXZZvwXtdj3wu30v8EL2tfdGWh7eYhy715fbcMWGBsF6gDpUjwjF+ihfezYZe9FcmA/8neSNJgzoXO4NwNMNBUKCJcWt2y8gZUhLzTvVf103oKqu/bzHQTihcU6RMLU1HVRbr6H3qeck26pDcSqk/ZRJld0WFtx9A+pVD1gzEwCLQ4BCmVMoMMFdyXVH4k0boxjckvIZUUeLpQeGVUj4xb86n48pTZco5AETmRqqVfGK31WjFY9m83+QgwqMCZR9qFMMVJPeyE8tpoqQXhxfJrNyHITgye/EPtVS5s0iLfWq/Qe9jHvIfKa5vlXm5kxyMBHacD07QrZqh4sy8vGPmJfwY8BmhcbeGjqx9yANX5O0KjVJlk2NtJ1yFd4PcMd4VJuVB3hMUPQwP6eZSfe+gj2n2epnL+KesaL+OgD8YRFg9ZyBPqz6VaHfSdLw5e/vmhLzLyiZkYT9K5/7UqFm/7W/VpMselZV7O1sYojIbWeIOcHtJ9wqeOQ20nl6phQJVyKLOY7ApOsLoqYfzkZoG/CrgYKheY20niiwqH+qG6V/rMGaM4RCfpI/2bmWJkhlxyuv+20yMWzRad3H1WpfgxMF0jz4OA3QQKxB+dF/+/50mpveJs7Plgfm3zH/whl+gfOAX6JqGJBFfCA7XUwxDW4EMyGnYj6YMbv3iTHxA30KpA//KmpdF+x6qCP3kbrBHLpNcCk8fqqP6xwojophid/lTieOyEgNL8iGTRpF+XICi+CJNFrIpGdONrmXcGMql2c28V3azX1NGscmXUZstNp0ckw9IuxJovgATgiOTUM+QLDjq2QVCP3Jxol+klK+0mHUGqVCMeQaEb/aG71LbPQNLNBk8SzT59QbBiCOgc6zVx0TtYoLNCbL0hMrocGAHZw1W82y24uAPuDJKMfyVEWJ4HBn6sonzeag1M2WuGJVJJ+o+QuxE1wXaPIYfyzWc6T3741TIc7gl669YjMvdbGxtC57/bjX+heWJyedxlbL9mtoPEApfWsYxrVRGodGXgeYf365HGhU6wjRU4Nq3Tkg/ptEbfPYDv1E83cEwc9UE7yGFXucQZbfggrxrLttPMNABX6ZWnsNOIiuQcolxcebjpBjDQJoE9GD+4XKFXCLcd7gMaS9EOgCig5o9sk6uC2D3jJWVlgCiZoFns2tqW8x2EvJFCPor4hsAgNqMEiyyLwxEiV8EYEnx1uFxxjF4irTY/TzKF0msPNpBuplra26qefrAcqXYgXH2bdx1jnfH7DRMijz41XcDcqWi+VR14SuSdoqT4WYHBJE8f0U9Ng8TPSvk9QLed8eFJmpirDRYTvXkJ9DJP9E39wvAH/W+6cVM//qCENyU=",
"publicKeyUri":"pubkey"}</pre>


== Version 1 and 2 ==
== Version 1 and 2 ==
Line 260: Line 378:


Like Version 2, except the URLs in the '''keyring''' hash are now relative to the symmetric key's URL.
Like Version 2, except the URLs in the '''keyring''' hash are now relative to the symmetric key's URL.
=== Example ===
<pre>{"keyring":{"../keys/pubkey":{"wrapped":"P3nQWhQtUHpUpdzI8RD0DkW3OSnNcrcOGhWBM6p7a9imIJ3K8RpsEVnhCIBmkprg40lddRgH/o+vDi2jYovxkFKUzM3izLDa016JZud7GWlx50WNpxaWmst76DlXtjnLDK44tJw2UzRJI6jKq/k5rm8anUNrBYWqS7v97+OiHG4viFI9IQCvRrCY0ow+Z37NppoArW/kbna8Dl4UQAkVlQNOgTjvWW6BtiXd3HN7iA4LFgGTJCuQhfMAZel0fRq9vkW6XCZr6OJqGeEXbfmkfwAMQ3S0THCeki6ejAhiGW70OXF+gBqBrYvTiqyVXuRNKRY6TtH+9g8Vb9aj2O6F/w==",
                              "hmac":"682bac23003ceaf660391b8342bb423f2e52200b238eacb9640dc39e69c72a72"}}}</pre>


== Version 2  ==
== Version 2  ==
Rnewman
canmove, Confirmed users
640

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/240714...338575"