View source for Raindrop/RaindropDocumentModel

== Introduction ==

Raindrop works with a number of conceptual entities, such as a 'message', 'identity', 'extension', 'account', etc.  The set of entities is informal and can be extended at any time.  These items are known as 'raindrop content'.  For example, a single conceptual 'message', or a single 'identity' is a single item of 'raindrop content', regardless of how that content is expressed. Every item of raindrop content has a 'raindrop key' - this key is an object that uniquely identifies the content item.  How this key is chosen depends on the 'provider' of the content item, but once created the raindrop key can be treated as an opaque blob - see below for more details.

A 'schema' is a definition of a set of fields and their meanings for a particular item of raindrop content.  A schema could almost be seen as an 'interface for data' - it is a 'contract' between the producer and consumer of data about what that data means. A 'schema instance' is a particular set of fields conforming to a schema which describe a particular item of raindrop content.  Schema instances are generally provided by raindrop extensions.  For example, we may define a schema ''rd.msg.body'' which defines the body and envelope data of an abstract message, and a schema called ''rd.tags'' which defines tags for any item.  Raindrop extensions will be able to create instances of these schemas for different items of content (eg, to define the tags for a specific message or identity.)  A schema instance must be thought of as a 'set of fields' rather than as any particular storage mechanism - in particular, we will try and avoid equating a schema instance with 'couch document' even if a 1:1 relationship exists - just think 'set of fields'. 

A single item of content is defined as the set of all schemas provided for a single 'raindrop key'.  As a result, an item of raindrop content does not have the concept of a 'type', only the concept of what schemas it makes available.  In other words, an item of raindrop content is not a 'message' or an 'identity' - it is just that some raindrop content items provide 'message' related schemas while others provide 'identity' related ones.  There is nothing in the model to prevent a single item of raindrop content providing both identity and message related schemas, although that probably doesn't make much sense.  Some items of raindrop content, such as account definitions or extensions, will provide neither message nor identity related schemas.

The intention is that our document model and API abstract away the storage of schemas - conceptually, all parts of raindrop deal only with the concept of reading and writing particular schema instances for particular 'raindrop keys' rather than dealing with the concept of couch documents.  While this abstraction makes sense for a number of reasons, the primary reason is that we want to reserve the right to reconsider our storage model based on our experiences as the size of the data grows.

== Storage of schemas ==

Each schema instance is written to its own couch document.  As a result, many couch documents combine to describe a single item of content.  Each couch documents holds the following fields:

* rd_key - the 'raindrop key' for the item.  This is a normal json object.

* rd_schema_id - the identity of the schema definition this document holds.

* rd_ext_id - the identity of the extension that created the schema

* rd_schema_confidence - the 'confidence' of the fields in this schema instance.

The fields defined by the schema are all stored directly in the couch document; this is the primary reason all special raindrop attributes begin with 'rd_'.  The _id for the couch document is built from the rd_key, rd_schema_id and rd_ext_id fields.  As a result each extension can currently only emit a single schema instance for an item of content.  This is likely to be revised once we deal with the concept of 'confidence' in more detail - see below.

Some other fields are maintained

* rd_source - the "source document" and revision that caused this schema instance to be created.  Following the rd_source attribute backwards through items will allow you to create a DAG of extension points that processed this message.

* rd_megaview_expandable - a list of field names that are able to be 'expanded' by the megaview.  In general this will list fields which are arrays of simple values, such as 'tags' and 'contacts'.  The value of this field is part of the schema definition and will automatically be set by the framework.  Currently the field and schema names are hard-coded, so Python code needs to change when a new schema is introduced with this requirement, but ultimately we expect the schema definitions themselves, including this type of metadata about the schema, validation functions etc, to live in couch docs and thus be truly dynamic.  See [[Raindrop/Megaview]] for more information.

== Backend extensions ==

Extensions themselves are also managed via this schema mechanism.  A special ''rd.core.ext'' schema exists to describe extensions and includes a field for their source-code.  Thus, new extensions can be created simply by creating a new schema item in the database with all the relevant fields.  This schema also includes the 'source schema IDs' they are interested in (ie, the ID of the schemas they take as input).  The back-end will create a separate 'work queue' for each extension, and as documents are modified in the database, the back-end will call all the extensions which have declared an interest in the schema.

The extensions themselves will generally create new schema instances.  In most cases they will emit new schema instances for the same item of raindrop content - for example, the 'conversation' extension point will generally emit a new ''rd.msg.conversation'' schema for the same message (ie, a new schema instance is written for an existing item of raindrop content).  However, these extensions are free to create anything they like - the conversation extension may also emit ''rd.msg.conversation'' schemas for new raindrop content items, thereby springing an entire new item of raindrop content into life.

As a result of this model there is no formal DAG for items of raindrop content - extensions declare the schema IDs they depend on, but not the schema IDs they create.  However, for any single item of raindrop content a DAG could be deduced.  So while a DAG must exist (or the queues would never finish), it can't be predicted ahead of time.  The 'viz-raindrop.py' script is capable of analyzing a database and creating a pretty-picture of the DAG it found.

== Multiple schema instances, confidences and aggregation. ==

A key requirement for raindrop is extensibility and schemas provide the key mechanism to achieve this.  One canonical example used by the raindrop team is the ability to override the 'from' field for an item.  For example, consider an email message which arrives, so the 'from' is recorded as being the email address which sent the message.  We would like to enable an extension to notice that this address is actually a special address (eg, an auto-generated mail on behalf of a real person) and change the 'from' value which is shown by the front-end application.  In our functional model, this extension is *not* free to simply modify the 'from' field which was initially written - instead some other mechanism to override that value must be provided.

This will be implemented using the concept of 'schema confidences'.  In the example above, there will be multiple ''rd.msg.body'' schemas specifying the 'from' field, but the 'from' field specified in the email address will have a 'lower confidence' than the from field provided by the extension point.  As a result, only the 'from' field with the highest confidence will be chosen.

An unfortunate complication of this is that not all schemas need to be treated this way.  In the example above, the front-end only wants to deal with a single ''rd.msg.body'' schema, built by using the highest confidence values and discarding the lower confidence values.  Further, even if an extension wants to override the 'from' field in the ''rd.msg.body'' schema, they may not want to override *all* fields, so we may need a mechanism which allows a 'smart merging' of only certain fields instead of the entire schema.

Other complications can be demonstrated by considering a schema trying to represent simple 'tags' - for example, #tags on twitter.  In this model we want a simple aggregation of these schemas, not a 'highest score wins' model, as these extensions are generally *contributing* to the complete set of tags, not overriding each other.  A further complication with this aggregation model is that the model needs to handle 'delete' as well as 'add' operations.  For example, consider a twitter message with a #tag.  An extension may emit a schema with this tag, however, if the user then wants to *remove* that tag from the message, our model must allow for that removal without modifying the original twitter message.  In other words, the schema emitted by the front-end in response to a user's tag operation must allow for tags to both be contributed to the final set, and remove specific items from the final set.

This requirement has largely been ignored in the current implementation.  The rd_schema_confidence field is defined but nothing writes to it.  Most code dealing with schemas currently just takes the very first instance of a schema that exists, ignoring the fact there may be others available.  '''We need to think about this more, and quickly''', as it is fairly fundamental.  Current thinking is that the couchdb 'list and show' features may be able to help us out here.

== raindrop keys ==

While the 'raindrop key' for an item is generally opaque, it must remain somewhat transparent for raw content providers.  These content providers will need to informally agree on the key format for certain items to help identify the same item of content from different providers - for example, an email message will have a key based on the message-id header, and if all providers of emails use the same key format, we will be able to detect and handle the same email message coming from different providers.

For another example of where keys are not quite opaque, consider the 'conversations' extension point - it has the requirement of stitching a conversation thread together from email messages, but some referenced messages may not exist on the server.  This extension point does know that the 'raindrop key' for an email message is of the form <pre>['email', message-id-header]</pre>, so it is capable of generating ''rd.msg.conversation'' schema records for these missing messages.  The extension is therefore springing new items of raindrop content into life - these are items without a ''rd.msg.body'' schema, but in all other ways are now bona-fide 'messages'.  Should this message later appear on the IMAP server a ''rd.msg.body'' schema would be created for the content item at that point.

== Views ==

The model described above allows for a single well-crafted view to slice and dice this data in various forms.

The megaview emits the schema id, field name and field value as the key for all content items.  Thus, you can determine all content items that hold a specific value for a specific field in a specific schema.  This view has a simple reduce function meaning that by using an appropriate group_level, you can determine which items define the field regardless of value, etc.

As an added bonus, the mega-view also emits a pseudo-schema called 'rd/core/content'.  This schema emits the values of the various 'rd_*' fields, meaning this view is also suitable to use as a meta-view.  For example, to find all documents which emit a particular schema, you can query for the key ['rd/core/content', 'schema_id', schema_name], or to query all items with a specific rd_key value (ie, all known schemas for a single raindrop content item), you could query for ['rd/core/content', 'key', rd_key_value].  Some of these values are concatenated, allowing you to specify for all schema items with a specific key, etc - see the megaview source for details.

The key advantage of this model is that there is no requirement to add custom views for each extension point, which is important given couchdb's view architecture; rebuilding a view index from scratch over a large database can take a very significant amount of time and resources.

The key downsides to this model are the complexity involved in learning to use the megaview, the huge amount of disk-space this view takes on disk and the lack of flexibility in performing queries which aren't able to be expressed by the megaview.