Confirmed users
358
edits
Identity/AttachedServices/StorageServerProtocol: Difference between revisions
Jump to navigation
Jump to search
Identity/AttachedServices/StorageServerProtocol (view source)
Revision as of 13:46, 18 June 2013
, 18 June 2013no edit summary
No edit summary |
No edit summary |
||
| Line 12: | Line 12: | ||
More details at [[Identity/CryptoIdeas/05-Queue-Sync]]. | More details at [[Identity/CryptoIdeas/05-Queue-Sync]]. | ||
Data is stored in independent named '''collections'''. A collection is a key-value store mapping keys to '''records'''. Each | Data is stored in independent named '''collections'''. A collection is a key-value store mapping keys to '''records'''. Each collection has a monotonically-increasing '''sequence number''' which is incremented whenever a record is changed, and provides the ability to request all '''changes''' since a given sequence number. | ||
collection has a monotonically-increasing '''sequence number''' which is incremented whenever a record is changed, and provides the | |||
ability to request all '''changes''' since a given sequence number. | |||
| Line 25: | Line 21: | ||
<tr><td>name</td><td>urlsafe string, 64 bytes</td><td>A unique identifier for this collection amongt all the user's data. Collection | <tr><td>name</td><td>urlsafe string, 64 bytes</td><td>A unique identifier for this collection amongt all the user's data. Collection | ||
names may only contain characters from the urlsafe-base64 alphabet (i.e. alphanumerics, underscore and hyphen).</td></tr> | names may only contain characters from the urlsafe-base64 alphabet (i.e. alphanumerics, underscore and hyphen).</td></tr> | ||
<tr><td>seqnum</td><td>integer, 8 bytes</td><td>A monotonically-increasing integer that is incremented with each change to the | <tr><td>seqnum</td><td>integer, 8 bytes</td><td>A monotonically-increasing integer that is incremented with each change to the | ||
contents of the collection.</td></tr> | contents of the collection.</td></tr> | ||
<tr><td>changeid</td><td>urlsafe string, XXX bytes</td><td>A hash that uniquely identifies the last change to this collection. It is | <tr><td>changeid</td><td>urlsafe string, XXX bytes</td><td>A hash that uniquely identifies the last change to this collection. It is | ||
derived from the new sequence number, the previous changeid, and the details of the change that was made.</td></tr> | derived from the new sequence number, the previous changeid, and the details of the change that was made.</td></tr> | ||
<tr><td>signature</td><td>urlsafe string, XXX bytes</td><td>A client-generated HMAC signature of the current changeid. Not used or | <tr><td>signature</td><td>urlsafe string, XXX bytes</td><td>A client-generated HMAC signature of the current changeid. Not used or | ||
verified by the server, since it doesn't have the secret key.</td></tr> | verified by the server, since it doesn't have the secret key.</td></tr> | ||
| Line 50: | Line 41: | ||
<tr><td>key</td><td>urlsafe string, 64 bytes</td><td>A unique identifier for this record within the collection. Keys may only contain | <tr><td>key</td><td>urlsafe string, 64 bytes</td><td>A unique identifier for this record within the collection. Keys may only contain | ||
characters from the urlsafe-base64 alphabet (i.e. alphanumerics, underscore and hyphen).</td></tr> | characters from the urlsafe-base64 alphabet (i.e. alphanumerics, underscore and hyphen).</td></tr> | ||
<tr><td>payload</td><td>urlsafe string, 256 KB</td><td>The value current stored in this record. Typically this would be encrypted and | <tr><td>payload</td><td>urlsafe string, 256 KB</td><td>The value current stored in this record. Typically this would be encrypted and | ||
signed by the client.</td></tr> | signed by the client.</td></tr> | ||
| Line 60: | Line 49: | ||
<tr><td>changeid</td><td>urlsafe string, XXX bytes</td><td>The collection-level changeid corresponding to the modification of this | <tr><td>changeid</td><td>urlsafe string, XXX bytes</td><td>The collection-level changeid corresponding to the modification of this | ||
record. It is derived from the new sequence number, the previous changeid, the record key, and the new record payload.</td></tr> | record. It is derived from the new sequence number, the previous changeid, the record key, and the new record payload.</td></tr> | ||
<tr><td>signature</td><td>urlsafe string, XXX bytes</td><td>A client-generated HMAC signature of the changeid for this record. Not | <tr><td>signature</td><td>urlsafe string, XXX bytes</td><td>A client-generated HMAC signature of the changeid for this record. Not | ||
used or verified by the server, since it doesn't have the secret key.</td></tr> | used or verified by the server, since it doesn't have the secret key.</td></tr> | ||
| Line 72: | Line 59: | ||
'''Change''' objects are identical to '''record''' objects, except their payload field may have the value NULL to indicate a deletion | '''Change''' objects are identical to '''record''' objects, except their payload field may have the value NULL to indicate a deletion | ||
rather than an update: | rather than an update: | ||
| Line 79: | Line 65: | ||
<tr><td>key</td><td>urlsafe string, 64 bytes</td><td>A unique identifier for the changed record within the collection. Keys may only | <tr><td>key</td><td>urlsafe string, 64 bytes</td><td>A unique identifier for the changed record within the collection. Keys may only | ||
contain characters from the urlsafe-base64 alphabet (i.e. alphanumerics, underscore and hyphen).</td></tr> | contain characters from the urlsafe-base64 alphabet (i.e. alphanumerics, underscore and hyphen).</td></tr> | ||
<tr><td>payload</td><td>urlsafe string or null, 256 KB</td><td>The new value to be stored in the record, or null if the record is to | <tr><td>payload</td><td>urlsafe string or null, 256 KB</td><td>The new value to be stored in the record, or null if the record is to | ||
be deleted. Typically this would be encrypted and signed by the client.</td></tr> | be deleted. Typically this would be encrypted and signed by the client.</td></tr> | ||
| Line 89: | Line 73: | ||
<tr><td>changeid</td><td>urlsafe string, XXX bytes</td><td>The new collection-level changeid corresponding to this change. It is | <tr><td>changeid</td><td>urlsafe string, XXX bytes</td><td>The new collection-level changeid corresponding to this change. It is | ||
derived from the new sequence number, the previous changeid, the record key, and the new record payload.</td></tr> | derived from the new sequence number, the previous changeid, the record key, and the new record payload.</td></tr> | ||
<tr><td>signature</td><td>urlsafe string, XXX bytes</td><td>A client-generated HMAC signature of the changeid. Not used or verified | <tr><td>signature</td><td>urlsafe string, XXX bytes</td><td>A client-generated HMAC signature of the changeid. Not used or verified | ||
by the server, since it doesn't have the secret key.</td></tr> | by the server, since it doesn't have the secret key.</td></tr> | ||
| Line 103: | Line 85: | ||
To access the storage service, a client device must authenticate by providing a BrowserID assertion and a Device ID. It will receive | To access the storage service, a client device must authenticate by providing a BrowserID assertion and a Device ID. It will receive | ||
in exchange: | in exchange: | ||
| Line 109: | Line 90: | ||
* a mapping of collection names to access URLs | * a mapping of collection names to access URLs | ||
You can think of this as establishing a "login session" with the server. Access requests for a specific collection should be directed | You can think of this as establishing a "login session" with the server. Access requests for a specific collection should then be directed | ||
to the appropriate URL. | to the appropriate URL. | ||
| Line 134: | Line 114: | ||
The user and device identity information is encoded in the hawk auth id, to avoid re-sending it on each request. The server may also | The user and device identity information is encoded in the hawk auth id, to avoid re-sending it on each request. The server may also | ||
include additional state in this value, depending on the implementation. It's opaque to the client. | include additional state in this value, depending on the implementation. It's opaque to the client. | ||
The collection-specific access URLs may include a unique identifier for the user, in order to improve RESTful-icity of the API. Or | The collection-specific access URLs may include a unique identifier for the user, in order to improve RESTful-icity of the API. Or | ||
they might point the client to a specific data-center which houses their write master for each collection. It's opaque to the client. | they might point the client to a specific data-center which houses their write master for each collection. It's opaque to the client. | ||
| Line 192: | Line 170: | ||
If there are a large number of records in the collection then the server may choose to paginate the result, returning only some of the | If there are a large number of records in the collection then the server may choose to paginate the result, returning only some of the | ||
records in the initial response. It will include the key "next" in the output to indicate that more records are available: | records in the initial response. It will include the key "next" in the output to indicate that more records are available: | ||
| Line 226: | Line 203: | ||
Records are always batched in lexicographic order of their keys, and clients are free to request an arbitrary key range using the | Records are always batched in lexicographic order of their keys, and clients are free to request an arbitrary key range using the | ||
'start' and 'end' parameters: | 'start' and 'end' parameters: | ||
| Line 242: | Line 218: | ||
Clients may also choose to batch their requests by using the 'limit' query parameter. As with server-driven batching, the output key | Clients may also choose to batch their requests by using the 'limit' query parameter. As with server-driven batching, the output key | ||
"next" will be used to indicate that more data is available: | "next" will be used to indicate that more data is available: | ||
| Line 272: | Line 247: | ||
Each server response will include an "ETag" header, formed from the combination of the current seqnum and changeid of the collection. | Each server response will include an "ETag" header, formed from the combination of the current seqnum and changeid of the collection. | ||
Clients can use this in combination with standard If-Match and If-None-Match headers to ensure that they're getting a consistent view | Clients can use this in combination with standard If-Match and If-None-Match headers to ensure that they're getting a consistent view | ||
of the collection: | of the collection: | ||
| Line 334: | Line 307: | ||
Get the sequence of changes that have been made to the collection. If the number of changes to be returned is small, they will be | Get the sequence of changes that have been made to the collection. If the number of changes to be returned is small, they will be | ||
returned all at once like so: | returned all at once like so: | ||
| Line 352: | Line 324: | ||
If there are a large number of changes to be fetched then the server may choose to paginate the result, returning only some of the | If there are a large number of changes to be fetched then the server may choose to paginate the result, returning only some of the | ||
changes in the initial request. It will include the key "next" in the output to indicate that more changes are available: | changes in the initial request. It will include the key "next" in the output to indicate that more changes are available: | ||
| Line 383: | Line 354: | ||
Records are always batched in sequence number order. Clients are free to request changes starting at an arbitrary sequence number, | Records are always batched in sequence number order. Clients are free to request changes starting at an arbitrary sequence number, | ||
which is useful for pulling in just the things that have changed since a previous sync. | which is useful for pulling in just the things that have changed since a previous sync. | ||
Clients may also choose to batch their requests by using the 'limit' query parameter. As with server-driven batching, the output key | Clients may also choose to batch their requests by using the 'limit' query parameter. As with server-driven batching, the output key | ||
"next" will be used to indicate that more data is available: | "next" will be used to indicate that more data is available: | ||
| Line 416: | Line 385: | ||
The server is not required to keep the full change history from seqnum zero, and may periodically compact and garbage-collection the | The server is not required to keep the full change history from seqnum zero, and may periodically compact and garbage-collection the | ||
stored data. If the client requests changes since a seqnum that is no longer known to the server, it will receive an error: | stored data. If the client requests changes since a seqnum that is no longer known to the server, it will receive an error: | ||
| Line 450: | Line 418: | ||
The server will apply each change in turn, checking that the seqnum and changeid hash chains are properly formed. If they are not | The server will apply each change in turn, checking that the seqnum and changeid hash chains are properly formed. If they are not | ||
then an error will be reported: | then an error will be reported: | ||
| Line 468: | Line 435: | ||
No content is returned in response to a POST. The client has already calculated the new seqnum and changeid for the collection, so | No content is returned in response to a POST. The client has already calculated the new seqnum and changeid for the collection, so | ||
there is no more useful information that the server can provide. | there is no more useful information that the server can provide. | ||
| Line 492: | Line 458: | ||
The server will check that the seqnum and changeid hash chains are properly formed before applying the change. If they are not then | The server will check that the seqnum and changeid hash chains are properly formed before applying the change. If they are not then | ||
an error will be reported: | an error will be reported: | ||
| Line 510: | Line 475: | ||
No content is returned in response to a POST. The client has already calculated the new seqnum and changeid for the collection, so | No content is returned in response to a POST. The client has already calculated the new seqnum and changeid for the collection, so | ||
there is no more useful information that the server can provide. | there is no more useful information that the server can provide. | ||