WebAPI/DataStore

Data Store API

This is the snapshot of a discussion between Mounir Lamouri, Thinker Lee, Gene Lian, Jonas Sicking and Hsin-Yi Tsai.
The work drafting happened on https://etherpad.mozilla.org/whatever-you-want

Use Cases

Allow an application to create data that can be shared with multiple other applications.
Allow multiple applications supply data to the same data store.
Support read-only stores like facebook contacts.
Support read/write stores like built-in contacts.
Support keeping an application-local cache of a data store. I.e. enable getting notified about changes to a data store so that the local cache can be kept up-to-date.
Enforce types of attributes (avoid to break other applications).

Why not...?

Why not use specific APIs for the use-cases like the existing Contacts and SMS/MMS APIs?

Here's an informative reply from Jonas Sicking on the dev-webapi list, included below:

The Contacts and MobileMessage APIs are richer in the sense that they support things like richer querying, like filtering, sorting and grouping.

However the Contacts and MobileMessage APIs has the severe shortcoming is that you are forced to live with the limitations of what querying capabilities those APIs have. Including the performance of those quering API.

We are *constantly* having to revise these APIs because it turns out that the querying capabilities aren't matching what our apps need. This is not a workable long term situation. And it's not even a workable short-term solution for 3rd party apps since we can't revise the APIs to support the capabilities that every 3rd party app developer needs.

This is why the DataStore API allows applications to synchronize data into a application-local cache. This cache can be stored/index/grouped/sorted in whatever format the application needs in order to support its UI. It even allows things like merge data from the MobilaMessage API and the Contacts API into a single location.

A very common reaction to this is "caching data in the application means duplicating the data!". This is true, but the current path we're on also causes a lot of duplication. Supporting all types of filtering, sorting and grouping like we do, forces us to create a lot of indexes. Indexes are effectively partial copies of the data. And we have to create those indexes whether they are actually used or not, because the API requires them.

By allowing applications to cache the data, only data that is actively needed by installed application is copied. And we enable applications to cache data in more formats than we could every think of and bake into the API.

This obviously doesn't mean that DataStore API will solve all of our problems. I suspect that something like the inter-app communication API will still be needed.

Why not use the Inter App Communication API?

WebAPI/Inter App Communication has more complicated security issues and is being worked at a lower priority.

Interface

 interface DataStore {
   // Returns the label of the DataSource.
   readonly attribute DOMString name;
   
   // Returns the origin of the DataSource (e.g., 'facebook.com').
   // This value is the manifest URL of the owner app.
   readonly attribute DOMString owner;
   
   // is readOnly a F(current_app, datastore) function? yes
   readonly attribute boolean readOnly;
   
   Promise<Object>  get(int id);
   Promise<void>    update(int id, Object obj);
   Promise<int>     add(Object obj);
   Promise<boolean> remove(int id);
   Promise<void>    clear();
   
   readonly attribute DOMString revisionId;
   
   attribute EventHandler onchange;
   Future<DataStoreChanges> getChanges(DOMString revisionId);
   
   // TODO: getAll(), getLength().
 };
 
 interface DataStoreChanges {
   readonly attribute DOMString revisionId;
   readonly attribute int[] addedIds;
   readonly attribute int[] updatedIds;
   readonly attribute int[] removedIds;
 }
 
 partial interface Navigator {
   Promise<sequence<DataStore>> getDataStores(DOMString name);
 };

Manifest

For the application that provides the datastore

 {
   ...
   datastores-owned: {
     "contacts": {
       "readonly": true,
       "description": "Facebook contacts",
     }
   },
   ...
 }

For the application that wants to access the datastore

 {
   ...
   datastores-access: {
     "contacts": {
       "access": "readonly",
       "description": ...
     }
   },
   ...
 }

Incremental Schema

DataStore is designed for sharing data among applications. Applications will make some assumptions on data types of attributes. If the data type of an attributes is not consistent among applications, applications may be broken. So, data types of attributes should be enforced. to define types of an attributes while attributes with a new path were found first time. In another word, once a new object was added to a data store, its tree of attributes will be traveled, and define the type of new found attributes with the type of their values.

For example, if the following object is the object been added to a data store.

 {
   SN: 123,
   name: "John Lin",
   info: {
     address: ".....",
     birth: Date(....),
   }
 }

Then, the types table of the data store is

 SN: Integer
 name: String
 info: object
 info address: String
 info birth: Date

Then, the following object is added.

 {
   SN: 123,
   name: "John Lin",
   info: {
     address: ".....",
     birth: Date(....),
     phone: "123456",
   }
 }

The types table should be

 SN: Integer
 name: String
 info: object
 info address: String
 info birth: Date
 info phone: String

Every time a new object was added to a data store, the types of attributes would be checked against the types table of the data store. The action of adding will be failed if the type of any attribute does not match the type defined in the types table.

Issues

* {name, owner, value} is a complicated key.
* UI: what to do when we have multiple access requests?
* What's happening if the central gets changes during the process of local updates?
* |addedIds|, |removedIds| and |updatedIds| arrays should be synchronized. For example, the ID of record that has been updated and removed should only show up in the |removedIds| array. Need to define the behaviours.
* Should all data stores with the same name share a schema?
* Enforcing types can be a footgun. What should a data provider do if it decides some key should have a different type?