Raindrop/WebApis
This outlines our current thinking and dilemmas regarding Raindrop Web APIs. We welcome all contributions and input into what we have described here.
High level goals
There are lots of reasons to have web based APIs to a messaging system like Raindrop. Here are some of them. Note that some are platform requirements, some are more product/application specific. We may need multiple APIs to support the different goals:
Modularity
Having well defined APIs would allow modularity between the various tiers of the system, from the message aggregation and back-end extension engine (currently CouchDB-based), to the conversation-aggregation layer (in the client code as of this writing), to the front-end UI code (in the client). Having APIs would allow us to be resilient to changes in CouchDB, to considering alternatives in any of the tiers, etc. It's just good software engineering.
Lighter weight clients
The current APIs require the client to do a lot of work, both in terms of processing power, and in terms of HTTP connections to the database. By moving some of that work behind an HTTP end-point, we should be better able to support clients on less capable platforms (e.g. phones), the performance of the client should improve, and in some installations we should be able to deploy web server technologies like caching layers to improve performance further.
Hackability
We have two APIs, right now, the direct CouchDB calls, which are complex due to the inherent complexity of the megaview (due to its genericity), and the very app-specific JavaScript APIs in the inflow code. Combined, it becomes harder to try new things like a non-JavaScript front-end.
Support and develop open standards
We want to help define open standards around messaging. We want to use current best practices in our domain - of which the only real contender is a REST based API accessed over HTTP. The API must allow for high-bandwidth low-latency applications.
Simplicity
The API should deal in the same concepts as the application rather than the details of the implementation; for example, messaging applications want to work with message related objects, not with database documents. The API should be simple enough to allow designers to easily experiment with concepts without needing them to understand all the technical details of the platform.
On simplicity...
While many of the above goals are self-explanatory, there are a number of aspects relating to the simplicity of the API that deserve further discussion.
Up in the sky, is it an application, is it a platform? No, it's a raindrop!
What exactly is a 'raindrop application'? The inflow application we ship with raindrop is intended to be just one of many possible raindrop applications. We expect people will innovate with their own applications, and while these are likely to be vaguely 'message based', they may deal with concepts and metaphors which are foreign to our inflow application.
This implies that we may need a sense of layering in the API itself, so each application can provide an application specific API built around some lower-level constructs provided by the platform. In other words, this API itself needs to be hugely extensible and probably based around a similar 'extension' model that raindrop itself uses.
Page based applications
Most raindrop applications are likely to be based in a browser, and is likely to have the concept of 'pages' (even if there is only 1 page). For applications which offer a number of pages (eg, the inflow application will have as many pages as necessary to show all your messages), it is very rare that a user will venture past the first couple of pages. If a user can't find what they want in those first few pages, they are likely to attempt a different 'vector' to locate what they want (such as searching, filtering, etc.)
A complication here is that exactly what is on a page depends on a number of factors - eg, the current filter being applied, even the size of the browser window. A simple user-interface action may drastically alter the items which appear on a given page.
A consequence of this is that:
- The API should, as much as possible, be based around the concept of pages.
- The API should be optimized around the idea that only the first couple of pages are likely to be hit. The first couple of pages must be able to be fetched and rendered quickly, whereas the user and application will be more tolerant of delays in rendering later pages.
- We need to think more about how the API can determine what a "page" really is, or a better model for implementing this requirement.
Extensible Applications
Raindrop has been designed to be highly extensible. Third parties can easily and simply add extensions to raindrop which add further attributes to contacts or messages. This introduces come conflict with the requirement to deal with 'application concepts' - we must design our API such that it works with entities like 'message' or 'conversation', while still allowing for extensions to change exactly what attributes and properties these entities possess.
The extensible 'api' mechanism
Like much of raindrop, individual APIs can be defined simply by adding new schema instances to the database. The schema rd.ext.api defines both the desired entry-point name(s) and the API implementation. As a result, it is possible for an 'extension' to also provide an API to the features that extension provides.
The API will be implemented by way of couchdb externals processes. Raindrop provides an "api-runner" which knows how to load API implementations from the database and expose them.
The 'inflow api'
One consequence of the above discussions is that each application is likely to require its own application specific API. This section describes the API associated with the inflow app.
As of the Maple milestone there is a start to the server API. Right now it is just focused on conversations, not contacts/identities.
This API is implemented using the mechanisms described above - look for the couch_docs/api files in the raindrop source tree.
Inflow API entry point: /raindrop/_api/inflow
Common GET URL parameters
- limit: the number of most recent messages to retrieve. These most recent messages are then converted to conversations so the actual amount of messages returned (on the list of conversations) can be larger than the amount specified by limit.
Specific API calls
- /raindrop/_api/inflow/conversations/personal: Gives most recent "direct" and "group" messages as specified by the rd.msg.recip-target schema. If the message is just to you or you are included in a message with other people it will show up in this list.
- /raindrop/_api/inflow/conversations/impersonal: Gives most recent "broadcast" and "notification" messages as specified by the rd.msg.recip-target schema. If it is a mailing list, or some email from a machine instead of a person it will show up in this list.
- /raindrop/_api/inflow/conversations/by_id Gives all messages for a given conversation_id. Takes these parameters:
- key: The conversation_id from a rd.msg.conversation schema. It should be formated as valid JSON (so surrounded by double-quotes) and URL encoded. For instance: %224AF4ABED.1050204%40example.com%22
Conversation Object
All the API calls above return an array of conversation objects. A Conversation object has the following properties:
- id: The conversation_id for the conversation. Taken from an rd.msg.conversation schema from the first message in the conversation.
- subject: The subject of the conversation. Taken from the first message's rd.msg.body's subject property.
- unread: The number of unread messages in the conversation. Taken from the first message's rd.msg.seen's schema, or the lack of that schema.
- people: An array of identity_ids that are associated with all the messages. Taken from all the messages, their rd.msg.body schema's from, to, cc or bcc properties.
- messages: An array of Message objects. The first one in the list is the message that started the conversation.
Message Object
Message objects inside a Conversation object have the following properties:
- schemas: an object whose properties are the schemas that make up the message. It does not include any "raw" schemas. Example schemas are "rd.msg.body" and "rd.msg.conversation".