WebAPI/DataStore

Data Store API

This is the snapshot of a discussion between Mounir Lamouri, Thinker Lee, Gene Lian, Jonas Sicking and Hsin-Yi Tsai.
The work drafting happened on https://etherpad.mozilla.org/whatever-you-want

Use Cases

Allow an application to create data that can be shared with multiple other applications.
Allow multiple applications supply data to the same data store.
Support read-only stores like facebook contacts.
Support read/write stores like built-in contacts.
Support keeping an application-local cache of a data store. I.e. enable getting notified about changes to a data store so that the local cache can be kept up-to-date.

Why not...?

Why not use specific APIs for the use-cases like the existing Contacts and SMS/MMS APIs?

Here's an informative reply from Jonas Sicking on the dev-webapi list, included below:

The Contacts and MobileMessage APIs are richer in the sense that they support things like richer querying, like filtering, sorting and grouping.

However the Contacts and MobileMessage APIs has the severe shortcoming is that you are forced to live with the limitations of what querying capabilities those APIs have. Including the performance of those quering API.

We are *constantly* having to revise these APIs because it turns out that the querying capabilities aren't matching what our apps need. This is not a workable long term situation. And it's not even a workable short-term solution for 3rd party apps since we can't revise the APIs to support the capabilities that every 3rd party app developer needs.

This is why the DataStore API allows applications to synchronize data into a application-local cache. This cache can be stored/index/grouped/sorted in whatever format the application needs in order to support its UI. It even allows things like merge data from the MobilaMessage API and the Contacts API into a single location.

A very common reaction to this is "caching data in the application means duplicating the data!". This is true, but the current path we're on also causes a lot of duplication. Supporting all types of filtering, sorting and grouping like we do, forces us to create a lot of indexes. Indexes are effectively partial copies of the data. And we have to create those indexes whether they are actually used or not, because the API requires them.

By allowing applications to cache the data, only data that is actively needed by installed application is copied. And we enable applications to cache data in more formats than we could every think of and bake into the API.

This obviously doesn't mean that DataStore API will solve all of our problems. I suspect that something like the inter-app communication API will still be needed.

Why not use the Inter App Communication API?

WebAPI/Inter App Communication has more complicated security issues and is being worked at a lower priority.

Interface

 interface DataStore : EventTarget {
   // Returns the label of the DataSource.
   readonly attribute DOMString name;
 
   // Returns the origin of the DataSource (e.g., 'facebook.com').
   // This value is the manifest URL of the owner app.
   readonly attribute DOMString owner;
   
   // is readOnly a F(current_app, datastore) function? yes
   readonly attribute boolean readOnly;
 
   // Promise<any>
   Promise get(unsigned long id);
 
   // Promise<any>
   Promise get(sequence<unsigned long> id);
 
   // Promise<void>
   Promise put(any obj, unsigned long id);
 
   // Promise<unsigned long>
   Promise add(any obj, optional unsigned long id);
 
   // Promise<boolean>
   Promise remove(unsigned long id);
 
   // Promise<void>
   Promise clear();
 
   readonly attribute DOMString revisionId;
 
   attribute EventHandler onchange;
 
   // Promise<DataStoreChanges>
   Promise getChanges(DOMString revisionId);
 
   // Promise<unsigned long>
   Promise getLength();
   
   DataStoreCursor sync(optional DOMString revisionId = "");
 };
 
 interface DataStoreCursor {
 
   // the DataStore
   readonly attribute DataStore store;
 
   // Promise<DataStoreTask>
   Promise next();
 
   void close();
 };
 
 enum DataStoreOperation {
   "add",
   "update",
   "remove",
   "clear",
   "done"
 };
 
 dictionary DataStoreTask {
   DOMString revisionId;  
 
   DataStoreOperation operation;
   unsigned long id;
   any data;
 };
 
 dictionary DataStoreChanges {
   DOMString revisionId;
   sequence<unsigned long> addedIds;
   sequence<unsigned long> updatedIds;
   sequence<unsigned long> removedIds;
 };
 
 dictionary DataStoreChangeEventInit : EventInit {
   DOMString revisionId = "";
   unsigned long id = 0;
   DOMString operation = "";
 };
 
 [Constructor(DOMString type, optional DataStoreChangeEventInit eventInitDict)]
 interface DataStoreChangeEvent : Event {
   readonly attribute DOMString revisionId;
   readonly attribute unsigned long id;
   readonly attribute DOMString operation;
 };

Manifest

For the application that provides the datastore

 {
   ...
   datastores-owned: {
     "contacts": {
       "readonly": true,
       "description": "Facebook contacts",
     }
   },
   ...
 }

For the application that wants to access the datastore

 {
   ...
   datastores-access: {
     "contacts": {
       "access": "readonly",
       "description": ...
     }
   },
   ...
 }

Revisions and changes

The revisionId is a UUID and it can be used to retrieve the delta between a particular revisionId and the current one. This operation is done using |Promise<DataStoreChanges> getChanges(DOMString revisionId);|. Note that the object DataStoreChanges contains only useful operations: for example, the ID of record that has been updated and removed only shows up in the |removedIds| array.

Examples

Basic operations

  // Here we retrieve the list of DataStores called 'contacts'.
  navigator.getDataStores("contacts").then(function(stores) {
    dump("DataStores called 'contacts': " + stores.length + "\n");
    
    if (!stores.length) return;
    
    dump("Current revisionID: " + stores[0].revisionId + "\n");
    
    // Retrieve an object from the first DataStore.
    stores[0].get(42).then(function(obj) {
      // ...
      
      // Update an object
      obj.nick = 'baku';
      stores[0].put(obj, 42).then(function(id) {
        // id == 42
        // ...
      }, function(error) {
        // something wrong happened. Error is a DOMError object.
      });
    });
    
    // Delete an object
    stores[0].remove(23).then(function(success) {
      if (success) {
        // The object has been deleted.
      } else {
        // Object 23 didn't exist.
      }
    });
    
    // Storing a new object
    stores[0].add({ "nick": "baku", "email" : "a@b.c" }).then(function(id) {
      // ...
    });
  });

Revisions

 // Let's assume that I have a revisionId stored somewhere (IndexedDb? LocalStorage?)
 var revisionId = 'b66e9248-5990-4ac3-9f5d-3cdeb02b337f';
 
 navigator.getDataStores('contacts').then(function(stores) {
   if (!stores.length) return;
   
   stores[0].getChanges(revisionId).then(function(delta) {
     if (revisionId == delta.revisionId) {
       // Nothing changed
     } else {
       dump("We lost: " + delta.removedIds + " contacts, but " + delta.addedIds + " have been added!\n");
       revisionId = delta.revisionId; // To store, somewhere.
     }
   });
 });

Sync

The synchronization of a DataStore with a 'private' app storage can be done using the 'sync' method. Calling this method, DataStore creates a DataStoreCursor that helps the app with the synchronization starting from scratch or for a valid revisionId. When a cursor is active, onchange events are not sent to the app. The sync operation can be terminated calling cursor.close(). If something changes in the DataStore when the cursor is synchronize the app, all the changes will be managed by the cursor as additional operation: this means that when the cursor completes its tasks, the app will be always in sync with the current revisionId of the DataStore.

The basic usage of the cursor is this:

 navagiator.getDataStores('contacts').then(functions(stores) {
   if (!stores.length) return;
 
   let cursor = stores[0].sync(/* a revisionId can be used here. If it's invalid it'll be ignored */);
 
   function cursorResolve(task) {
     // task.operation describes what the app has to do in order to be in sync with the current revision of this datastore.
     switch (task.operation) {
        case 'done':
          // No additional operation has to be done. When 'done' is received, the sync is completed and onchange events can be received again.
          dump("The current revision ID is: " + task.revisionId + "\n");
          return;
 
       case 'clear':
         // All the data you have are out-of-sync. Delete all of them.
         break;
 
       case 'add':
         // A new object has to be inserted
         dump("Adding id: " + task.id + " data: " + task.data + "\n");
         break;
 
       case 'update':
         // Something has to be updated
         dump("Updating id: " + task.id + " data: " + task.data + "\n");
         break;
 
       case 'remove':
         dump("Record: " + task.id + " must be removed\n");
         break;
     }
 
     cursor.next().then(cursorResolve);
   }
 
   // Cursor.next() always returns a promise.
   cursor.next().then(cursorResolve);
 });

Issues

* {name, owner, value} is a complicated key.
* UI: what to do when we have multiple access requests?
* Should all data stores with the same name share a schema?
* Enforcing types can be a footgun. What should a data provider do if it decides some key should have a different type?