Firefox OS/Vaani/Music

Proposal

Vaani will connect to music-controller IAC while Vaani receives a control commands, play, stop, pause, resume, previous, or next. No matter music app is opened or not, Vaani will open or connect to it at background and sends command to music app. Unlike mediacomms at system app, system app waits for playing state change to activate media widget at utility tray or lockscreen. Vaani uses music-controller in active mode.

In our UX spec, Vaani will send the following messages to music:

• if music is playing
• to play next song
• to play previous song
• to play all
• to stop/pause playing
• to resume playing
• who is artist (out of scope)
• what album is this (out of scope)

Music widget at Vaani

We will write a UI at Vaani to have music control widget. The state and information on top of it are updated with the change event through IAC.

IAC and future protocol

Currently, we only have IAC to communicate between apps. Music app should host IAC channel. If music app is not opened, Vaani can use this to open music app and send commands.

In the future, we should create an API for Vaani. Apps only need to say how many actions it supports.

Supported commands

We supports the following actions for music controlling use case:

• ListenAction
• StopAction
• SuspendAction
• SkipBackwardAction
• SkipForwardAction
• SearchAction

Extended schemata

We may extend some actions to meet our requirement:

1. Thing > Action > ControlAction > StopAction
2. Thing > Action > ControlAction > SkipForwardAction
3. Thing > Action > ControlAction > SkipBackwardAction

Proposal 1: JSON-LD in manifest

In proposal 1, An app who supports custom command should put the commands they want to register in manifest.webapp. The formats are like following section.

ListenAction

   "custom_commands": [{
       "@context": "http://schema.org",
       "@type": "ListenAction",
       "target": {
         "@type": "EntryPoint",
         "urlTemplate": “iac://music-controller",
         "actionApplication": {
           "@type": "SoftwareApplication",
           "operatingSystem": “Firefox OS"
         }
       }
     }]

StopAction

     "custom_commands": [{
       "@context": "http://url-to-extended-schema",
       "@type": "StopAction",
       "target": {
         "@type": "EntryPoint",
         "urlTemplate": “iac://music-controller",
         "actionApplication": {
           "@type": "SoftwareApplication",
           "operatingSystem": “Firefox OS"
         }
       }
     }]

SuspendAction

     "custom_commands": [{
       "@context": "http://schema.org",
       "@type": "SuspendAction",
       "target": {
         "@type": "EntryPoint",
         "urlTemplate": “iac://music-controller",
         "actionApplication": {
           "@type": "SoftwareApplication",
           "operatingSystem": “Firefox OS"
         }
       }
     }]

SkipBackwardAction

     "custom_commands": [{
       "@context": "http://url-to-extended-schema",
       "@type": "SkipBackwardAction",
       "target": {
         "@type": "EntryPoint",
         "urlTemplate": “iac://music-controller",
         "actionApplication": {
           "@type": "SoftwareApplication",
           "operatingSystem": “Firefox OS"
         }
       }
     }]

SkipForwardAction

     "custom_commands": [{
       "@context": "http://url-to-extended-schema",
       "@type": "SkipForwardAction",
       "target": {
         "@type": "EntryPoint",
         "urlTemplate": “iac://music-controller",
         "actionApplication": {
           "@type": "SoftwareApplication",
           "operatingSystem": “Firefox OS"
         }
       }
     }]

SearchAction

     "custom_commands": [{
       "@context": "http://schema.org/",
       "@type": "SearchAction",
       "object": ["MusicGroup", "MusicAlbum"],
       "target": {
         "@type": "EntryPoint",
         "urlTemplate": “iac://music-controller",
         "actionApplication": {
           "@type": "SoftwareApplication",
           "operatingSystem": “Firefox OS"
         }
       }
     }]

Proposal 2: not JSON-LD

Unlike proposal 1, proposal 2 is based on feature set instead of single action. We group actions which will be called with the same channel as a feature set. This design looks simpler and no JSON-LD or schema.org data introduced to manifest.webapp.

An app who supports custom command should put the following section in manifest.webapp, which is for IAC mechanism:

    "custom_commands": [{
       "iac-channel": "media-controller",
       "potentialAction": ["PlayAction", "StopAction"]
     }]

In the future, we should create an API for Vaani. Vaani can open app while it wants to send message to them. In this version, we don't need iac-channel field and let Vaani to open the app to predefined url. Like IAC handler, apps handle mozSetMessageHandler with vaani(TBD) message.

    "custom_commands": [{
       "feature-set": "media-controller",
       "(optional)name": "Music Player",
       "(optional)description": "Control your music playing with voice",
       "launch-path": "index.html",
       "potentialAction": ["PlayAction", "StopAction"]
     }]

Protocols for each command

Play music

For an app who wants to support Play music, it should handle the ListenAction. Music app will receive a ListenAction through IAC. The message may look like:

     {
       "@context": "http://schema.org",
       "@type": "ListenAction"
     }

And the app have to send back a message like:

     {
       "@context": "http://schema.org",
       "@type": "ListenAction",
       "actionStatus": "CompletedActionStatus" | "FailedActionStatus"
     }

Stop

For an app who wants to support Stop, it should handle the StopAction. Music app will receive a StopAction through IAC. The message may look like:

     {
       "@context": "http://schema.org",
       "@type": "StopAction"
     }

And the app have to send back a message like:

     {
       "@context": "http://schema.org",
       "@type": "StopAction",
       "actionStatus": "CompletedActionStatus" | "FailedActionStatus"
     }

Pause

For an app who wants to support Pause, it should handle the SuspendAction. Music app will receive a SuspendAction through IAC. The message may look like:

     {
       "@context": "http://schema.org",
       "@type": "SuspendAction"
     }

And the app have to send back a message like:

     {
       "@context": "http://schema.org",
       "@type": "SuspendAction",
       "actionStatus": "CompletedActionStatus" | "FailedActionStatus"
     }

Previous

For an app who wants to support Previous command, it should handle the SkipBackwardAction. Music app will receive a SkipBackwardAction through IAC. The message may look like:

     {
       "@context": "http://schema.org",
       "@type": "SkipBackwardAction"
     }

And the app have to send back a message like:

     {
       "@context": "http://schema.org",
       "@type": "SkipBackwardAction",
       "actionStatus": "CompletedActionStatus" | "FailedActionStatus"
     }

Next

For an app who wants to support Next command, it should handle the SkipForwardAction. Music app will receive a SkipForwardAction through IAC. The message may look like:

     {
       "@context": "http://schema.org",
       "@type": "SkipForwardAction"
     }

And the app have to send back a message like:

     {
       "@context": "http://schema.org",
       "@type": "SkipForwardAction",
       "actionStatus": "CompletedActionStatus" | "FailedActionStatus"
     }

Who is the artist / What album is this

If we want to query "Who is the artist", we may send a SearchAction with object equal to MusicGroup and without query to query the artist of current playing. If we want to query "What album is this", we may send a SearchAction with object equal to MusicAlbum and without query to query the album of current playing.

While handling it, music app will receive a SearchAction through IAC. The message may look like:

     {
       "@context": "http://schema.org",
       "@type": "SearchAction",
       "object": ["MusicGroup"] | ["MusicRecording"]
     }

And music app may send a message like below back:

     {
       "@context": "http://schema.org",
       "@type": "SearchAction"
       "actionStatus": "CompletedActionStatus" | "FailedActionStatus",
       "result": [{
           @type: "MusicGroup",
           name: "group name"
       }]
     }

FAQ

Where to have the apps expose its actions ?

We use JSON-LD to describe our commands. And in current phase, we may put those commands at manifest.webapp due to the requirement of preloading the commands before the applications are launched. In web pages, JSON-LD is usually embedded in script as this example shows, which is also part of w3c spec. This is also a In contrast to web page, the metadata of an app may be in manifest.webapp. It makes sense to put those command at manifest.webapp.

There is a bug 1178491 about JSON-LD and "pin the web" feature is also use JSON-LD, we should discuss with them.

Are there any solutions that can meet both conditions? This is also the question we are searching for a good answer.

Use ListenAction for "Play Music" command ?

For the control actions like pause for SuspendAction, Stop for StopAction we have our all actions under ControlAction. Should we make the action of "Play Music" under ControlAction too ?