WebAPI/DataStore

Data Store API

This is the snapshot of a discussion between Mounir Lamouri, Thinker Lee, Gene Lian, Jonas Sicking and Hsin-Yi Tsai.
The work drafting happened on https://etherpad.mozilla.org/whatever-you-want

Use Cases

Allow an application to create data that can be shared with multiple other applications.
Allow multiple applications supply data to the same data store.
Support read-only stores like facebook contacts.
Support read/write stores like built-in contacts.
Support keeping an application-local cache of a data store. I.e. enable getting notified about changes to a data store so that the local cache can be kept up-to-date.

Why not...?

Why not use specific APIs for the use-cases like the existing Contacts and SMS/MMS APIs?

Here's an informative reply from Jonas Sicking on the dev-webapi list, included below:

The Contacts and MobileMessage APIs are richer in the sense that they support things like richer querying, like filtering, sorting and grouping.

However the Contacts and MobileMessage APIs has the severe shortcoming is that you are forced to live with the limitations of what querying capabilities those APIs have. Including the performance of those quering API.

We are *constantly* having to revise these APIs because it turns out that the querying capabilities aren't matching what our apps need. This is not a workable long term situation. And it's not even a workable short-term solution for 3rd party apps since we can't revise the APIs to support the capabilities that every 3rd party app developer needs.

This is why the DataStore API allows applications to synchronize data into a application-local cache. This cache can be stored/index/grouped/sorted in whatever format the application needs in order to support its UI. It even allows things like merge data from the MobilaMessage API and the Contacts API into a single location.

A very common reaction to this is "caching data in the application means duplicating the data!". This is true, but the current path we're on also causes a lot of duplication. Supporting all types of filtering, sorting and grouping like we do, forces us to create a lot of indexes. Indexes are effectively partial copies of the data. And we have to create those indexes whether they are actually used or not, because the API requires them.

By allowing applications to cache the data, only data that is actively needed by installed application is copied. And we enable applications to cache data in more formats than we could every think of and bake into the API.

This obviously doesn't mean that DataStore API will solve all of our problems. I suspect that something like the inter-app communication API will still be needed.

Why not use the Inter App Communication API?

WebAPI/Inter App Communication has more complicated security issues and is being worked at a lower priority.

Interface

 interface DataStore : EventTarget {
   // Returns the label of the DataSource.
   readonly attribute DOMString name;
   
   // Returns the origin of the DataSource (e.g., 'facebook.com').
   // This value is the manifest URL of the owner app.
   readonly attribute DOMString owner;
   
   // is readOnly a F(current_app, datastore) function? yes
   readonly attribute boolean readOnly;
   
   Promise<Object>  get(int id);
   Promise<void>    update(int id, Object obj);
   Promise<int>     add(Object obj);
   Promise<boolean> remove(int id);
   Promise<void>    clear();
   
   readonly attribute DOMString revisionId;
   
   attribute EventHandler onchange;
   Promise<DataStoreChanges> getChanges(DOMString revisionId);
   
   // TODO: getAll(), getLength().
 };
 
 interface DataStoreChanges {
   readonly attribute DOMString revisionId;
   readonly attribute int[] addedIds;
   readonly attribute int[] updatedIds;
   readonly attribute int[] removedIds;
 }
 
 partial interface Navigator {
   Promise<sequence<DataStore>> getDataStores(DOMString name);
 };

Manifest

For the application that provides the datastore

 {
   ...
   datastores-owned: {
     "contacts": {
       "readonly": true,
       "description": "Facebook contacts",
     }
   },
   ...
 }

For the application that wants to access the datastore

 {
   ...
   datastores-access: {
     "contacts": {
       "access": "readonly",
       "description": ...
     }
   },
   ...
 }

Revisions and changes

The revisionId is a UUID and it can be used to retrieve the delta between a particular revisionId and the current one. This operation is done using |Promise<DataStoreChanges> getChanges(DOMString revisionId);|. Note that the object DataStoreChanges contains only useful operations: for example, the ID of record that has been updated and removed only shows up in the |removedIds| array.

Examples

Basic operations

  // Here we retrieve the list of DataStores called 'contacts'.
  navigator.getDataStores("contacts").then(function(stores) {
    dump("DataStores called 'contacts': " + stores.length + "\n");
    
    if (!stores.length) return;
    
    dump("Current revisionID: " + stores[0].revisionId + "\n");
    
    // Retrieve an object from the first DataStore.
    stores[0].get(42).then(function(obj) {
      // ...
      
      // Update an object
      obj.nick = 'baku';
      stores[0].update(42, obj).then(function(id) {
        // id == 42
        // ...
      }, function(error) {
        // something wrong happened. Error is a DOMError object.
      });
    });
    
    // Delete an object
    stores[0].remove(23).then(function(success) {
      if (success) {
        // The object has been deleted.
      } else {
        // Object 23 didn't exist.
      }
    });

    // Storing a new object
    stores[0].add({ "nick": "baku", "email" : "a@b.c" }).then(function(id) {
      // ...
    });
  });

Revisions

 // Let's assume that I have a revisionId stored somewhere (IndexedDb? LocalStorage?)
 var revisionId = 'b66e9248-5990-4ac3-9f5d-3cdeb02b337f';
 
 navigator.getDataStores('contacts').then(function(stores) {
   if (!stores.length) return;
   
   stores[0].getChanges(revisionId).then(function(delta) {
     // TODO
   });
 });

Issues

* {name, owner, value} is a complicated key.
* UI: what to do when we have multiple access requests?
* What's happening if the central gets changes during the process of local updates?
* Should all data stores with the same name share a schema?
* Enforcing types can be a footgun. What should a data provider do if it decides some key should have a different type?