View source for Raindrop/Extensions/Data Miners

{{draft}}

Data Mining messages is a how we provide more value to the user than just displaying the pure text of a message.  By looking for patterns in our conversations we can help people get more information and do more faster.

Almost every data mining extension works in this way:

# Find a pattern in a message
# Possibly call out to an external service for more meta-data about the pattern
# Create (emit) a document about the information

Here are some examples of different services:

== Photo Services ==

When people share links to their online photographs they are really sharing the photos themselves.  Our goal is to get a thumbnail of the image, a title, location, and any other data that might be interesting.  

=== Flickr ===

The Flickr data mining extension scans messages for flickr URLs it recognizes.  Currently this handles both flickr.com URLs and the shortened flic.kr URLs.

Here you can see the code for when the [http://hg.mozilla.org/labs/raindrop/rev/ba355dcf8e9e Flickr Data Miner] was first added and here when the [http://hg.mozilla.org/labs/raindrop/rev/e320adcd2ca7 flick.kr urls] where added as well.

Here are the latest versions of the Flickr data mining extension.
* [http://hg.mozilla.org/labs/raindrop/file/tip/couch_docs/wq/rd.ext.core.msg-body-quoted-to-flickr.json JSON Registration File]
* [http://hg.mozilla.org/labs/raindrop/file/tip/couch_docs/wq/rd.ext.core.msg-body-quoted-to-flickr.py Data Mining Extension]

=== TweetPhoto ===

''Minimal TweetPhoto support has been added, [http://hg.mozilla.org/labs/raindrop/rev/6df827ad66b1]''

* API: http://tweetphoto.com/developer
** GET PHOTO: http://groups.google.com/group/tweetphoto/web/get-photo-details
* Example
** http://tweetphoto.com/9298206
** XML: http://tweetphotoapi.com/api/tpapi.svc/photos/9298206
** JSON: http://tweetphotoapi.com/api/tpapi.svc/json/photos/9298206

=== Twitpic ===

: ''Minimal Twitpic support has been added [http://hg.mozilla.org/labs/raindrop/rev/6df827ad66b1]''

* API: http://twitpic.com/api.do#thumbnails
* Example
** There isn't a method for getting a title other than grabbing the HTML and getting the actual title from the page
** http://twitpic.com/y8imi
*** THUMB: http://twitpic.com/show/thumb/y8imi
*** MINI: http://twitpic.com/show/mini/y8imi

== Link Shorteners ==

With the rise of [http://twitter.com/ Twitter] there are a growing number of link shortening services.  These services are excellent for saving on the number of characters in your messages however they hide the real location of links and can make messages harder to understand.

In Raindrop we expand these links to get their full URL, title, and any other information which might be interesting.  This allows us to give the person more information about the link before clicking on it.

Currently Raindrop cannot data mine the full URL of shortened links which are expanded.  There a limitation where a shortened link which points to youtube, flickr, or other another service will only show up as the full URL and will not be data mined further so a thumbnail or title can be retrieved.

=== Bit.ly ===

The [http://biy.ly bit.ly] URL expander uses the [http://code.google.com/p/bitly-api/wiki/ApiDocumentation API] for expanding shortened links found in messages. 

Here's the initial commit of the [http://hg.mozilla.org/labs/raindrop/rev/4ec54fd894d1 bit.ly extension]

The extension does the following:

# Search for bit.ly links in messages
# Call out to the bit.ly API for the link title and other meta-data
# Create (emit) a document describing the expanded link

Here are the latest versions of the extension:
* [http://hg.mozilla.org/labs/raindrop/file/tip/couch_docs/wq/rd.ext.core.msg-body-quoted-to-bitly.json JSON Registration File]
* [http://hg.mozilla.org/labs/raindrop/file/tip/couch_docs/wq/rd.ext.core.msg-body-quoted-to-bitly.py Data Mining Extension]

=== retwt.me ===

[http://retwt.me/ retwt.me] has a nice and easy [http://retwt.me/-/about/api API] for expanding a link.  Without an API key you can quickly and easily get the expanded link version.  The expanded link does not include a title or any extra meta-data.

e.g.
  curl "http://api.retwt.me/expand.json?shortUrl=http://retwt.me/1LVxs"

=== is.gd ===

''An is.gd extension has not yet been implemented''

[http://is.gd is.gd] offers an [http://is.gd/api_info.php API] via the [http://is.gd/tech.php tech page] however the "expand" API method is to follow the links 301 redirect.

=== BudURL ===

''A BudURL extension has not yet been implemented, do you want to take it on?''

[http://budurl.com/ budurl.com] offers an [http://budurl.com/page/api API] for  expanding shortened links found in messages.

=== tr.im ===

''A tr.im extension has not yet been implemented''

[http://tr.im/ tr.im] offers an [http://tr.im/website/api API] for expanding shortened links found in messages.  An API key is required to make most calls.

=== su.pr ===

''An su.pr extension has not yet been implemented''

[http://su.pr/ su.pr] or [http://www.stumbleupon.com/ StumbleUpon] offers an [http://www.stumbleupon.com/developers/Supr:API_documentation/ API] for expanding shortened links.

  curl "http://su.pr/api/expand?shortUrl=http://su.pr/$HASH"

  {
   "errorCode":0,
   "errorMessage":"",
   "results": {
              "$HASH": { 
                         "longUrl":"$HTTP"
                         }
              },
   "statusCode":"OK"
  }

== Video Services ==

The intention of the Video service is to grab a thumbnail, title, location, and inline video player of a video link used in a message.  With this information we can create a much better video experience that allows people to view the video inline.

=== YouTube ===

Here are the latest versions of the YouTube extension:
* [http://hg.mozilla.org/labs/raindrop/file/tip/couch_docs/wq/rd.ext.core.msg-hyperlink-to-youtubed.json JSON Registration File]
* [http://hg.mozilla.org/labs/raindrop/file/tip/couch_docs/wq/rd.ext.core.msg-hyperlink-to-youtubed.py Data Mining Extension]

=== Vimeo ===

Here's the initial commit of [http://hg.mozilla.org/labs/raindrop/rev/eabbf7f08184 Vimeo link support] into Raindrop.

Here are the latest extension files:
* [http://hg.mozilla.org/labs/raindrop/file/tip/couch_docs/wq/rd.ext.core.msg-body-quoted-to-vimeo.json JSON Registration File]
* [http://hg.mozilla.org/labs/raindrop/file/tip/couch_docs/wq/rd.ext.core.msg-body-quoted-to-vimeo.py Data Mining Extension]

== Links ==

Certain web sites offer simple APIs for retrieving more information about the products they offer.  We can optimize for many of these sites by creating extensions that find those links and grab interesting information from the sites to expand on the link given.

=== Mozilla Add-ons ===

:''This extension is a suggestion that has not been developed.  Try it out!''

Using the [[Update:Remora API Docs|AMO API]] Raindrop can retrieve additional information about an Add-on link being sent via email.

If someone sends a link like: https://addons.mozilla.org/en-US/firefox/addon/748

Raindrop can find that link and retrieve the extra meta data via the API by constructing a URL like this: https://services.addons.mozilla.org/en-US/firefox/api/1.3/addon/748

Now you'd just need to create a quick AMO UI extension helper to display the real title, image, and author; you could even offer an install button for a link sent.


=== Four Square ===

: ''This '''has''' been initially implemented in Raindrop''

[http://foursquare.com/ Four Square] has an [http://groups.google.com/group/foursquare-api/web/api-documentation API] which allows for simple lookups of venues

=== Gowalla ===

: ''Gowalla support has not been implemented in Raindrop''

[http://gowalla.com/ Gowalla] has [http://gowalla.com/blog/2010/02/announcing-the-gowalla-api/ announced an API] which could be used to retrieve information about gowalla links.