Audio Data API: Difference between revisions

Jump to navigation Jump to search

Audio Data API (view source)

Revision as of 07:22, 18 May 2013

3,014 bytes added ,  18 May 2013
m
→‎Defining an Enhanced API for Audio (Draft Recommendation)
 
(39 intermediate revisions by 11 users not shown)
Line 1: Line 1:
== Defining an Enhanced API for Audio (Draft Recommendation) ==
== Defining an Enhanced API for Audio (Draft Recommendation) ==
'''Note''': this API has been ''deprecated'' in favor of the [https://developer.mozilla.org/en-US/docs/Web_Audio_API Web Audio API] chosen by the W3C.


===== Abstract =====
===== Abstract =====
Line 18: Line 20:
* Thomas Saunders
* Thomas Saunders
* Ted Mielczarek
* Ted Mielczarek
== Standardization Note ==
Please note that this document describes a non-standard experimental API.  This API is considered deprecated and may not be supported in future releases.  The World Wide Web Consortium (W3C) has chartered the [http://www.w3.org/2011/audio/ Audio Working Group] to develop standardized audio API specifications, including [[Web Audio API]].  Please refer to the Audio Working Group website for further details.


== API Tutorial ==
== API Tutorial ==
Line 310: Line 316:
function audioAvailable(event) {
function audioAvailable(event) {
   // Write the current framebuffer
   // Write the current framebuffer
   var frameBuffer = event.frameBuffer;
   var frameBuffer = event.frameBuffer; // frameBuffer is Float32Array
   writeAudio(frameBuffer);
   writeAudio(frameBuffer);
}
}
Line 317: Line 323:
a1.addEventListener('loadedmetadata', loadedMetadata, false);
a1.addEventListener('loadedmetadata', loadedMetadata, false);


function writeAudio(audio) {
function writeAudio(audioBuffer) {
   buffers.push(audio);
  // audioBuffer is Float32Array
   buffers.push({buffer: audioBuffer, position: 0});


   // If there's buffered data, write that
   // If there's buffered data, write that
   while(buffers.length > 0) {
   while(buffers.length > 0) {
     var buffer = buffers.shift();
     var buffer = buffers[0].buffer;
     var written = a2.mozWriteAudio(buffer);
    var position = buffers[0].position;
     var written = a2.mozWriteAudio(buffer.subarray(position));
     // // If all data wasn't written, keep it in the buffers:
     // // If all data wasn't written, keep it in the buffers:
     if(written < buffer.length) {
     if(position + written < buffer.length) {
       buffers.unshift(buffer.slice(written));
       buffers[0].position = position + written;
       return;
       break;
     }
     }
    buffers.shift();
   }
   }
}
}
Line 374: Line 383:
         var currentWritePosition = 0;
         var currentWritePosition = 0;
         var prebufferSize = sampleRate / 2; // buffer 500ms
         var prebufferSize = sampleRate / 2; // buffer 500ms
         var tail = null;
         var tail = null, tailPosition;


         // The function called with regular interval to populate  
         // The function called with regular interval to populate  
Line 381: Line 390:
           var written;
           var written;
           // Check if some data was not written in previous attempts.
           // Check if some data was not written in previous attempts.
           if(tail) {
           if(tail) {
             written = audio.mozWriteAudio(tail);
             written = audio.mozWriteAudio(tail.subarray(tailPosition));
             currentWritePosition += written;
             currentWritePosition += written;
             if(written < tail.length) {
            tailPosition += written;
             if(tailPosition < tail.length) {
               // Not all the data was written, saving the tail...
               // Not all the data was written, saving the tail...
              tail = tail.slice(written);
               return; // ... and exit the function.
               return; // ... and exit the function.
             }
             }
Line 404: Line 413:
             if(written < soundData.length) {
             if(written < soundData.length) {
               // Not all the data was written, saving the tail.
               // Not all the data was written, saving the tail.
               tail = soundData.slice(written);
               tail = soundData;
              tailPosition = written;
             }
             }
             currentWritePosition += written;
             currentWritePosition += written;
Line 456: Line 466:
interface nsIDOMNotifyAudioAvailableEvent : nsIDOMEvent
interface nsIDOMNotifyAudioAvailableEvent : nsIDOMEvent
{
{
   // mozFrameBuffer is really a Float32Array
   // frameBuffer is really a Float32Array
   readonly attribute jsval  frameBuffer;
   readonly attribute jsval  frameBuffer;
   readonly attribute float  time;
   readonly attribute float  time;
Line 519: Line 529:
===== Compatibility with Audio Backends =====
===== Compatibility with Audio Backends =====


The current MozAudioAvailable implementation integrates with Mozilla's decoder abstract base classes, and therefore, any audio decoder which uses these base classes automatically dispatches MozAudioAvailable events.  At the time of writing, this includes the Ogg and WebM decoders but '''not''' the Wave decoder.
The current MozAudioAvailable implementation integrates with Mozilla's decoder abstract base classes, and therefore, any audio decoder which uses these base classes automatically dispatches MozAudioAvailable events.  At the time of writing, this includes the Ogg, WebM, and Wave decoders.


== Additional Resources ==
== Additional Resources ==
Line 539: Line 549:
=== JavaScript Audio Libraries ===
=== JavaScript Audio Libraries ===


* We have started work on a JavaScript library to make building audio web apps easier.  Details are [[Audio Data API JS Library|here]] and http://github.com/corbanbrook/dsp.js.
* We have started work on a JavaScript library to make building audio web apps easier.  Details are [[Audio Data API JS Library|here]] and https://github.com/corbanbrook/dsp.js.
* [http://github.com/corbanbrook/audionode.js audionode.js] acts as a javascript bridge between the Web Audio API and the Audio Data API allowing us to run the examples http://weare.buildingsky.net/processing/audionode.js/examples/index.html.
* [https://github.com/corbanbrook/audionode.js audionode.js] acts as a javascript bridge between the Web Audio API and the Audio Data API allowing us to run the examples http://weare.buildingsky.net/processing/audionode.js/examples/index.html.
* [http://github.com/notmasteryet/audiodata/ Audio Data API Objects] - An high level abstraction (and an usage example) of the Audio Data API.
* [https://github.com/notmasteryet/audiodata Audio Data API Objects] - An high level abstraction (and an usage example) of the Audio Data API.
* [http://github.com/bfirsh/dynamicaudio.js dynamicaudio.js] - An interface for writing audio with a Flash fall back for older browsers.
* [https://github.com/bfirsh/dynamicaudio.js dynamicaudio.js] - An interface for writing audio with a Flash fall back for older browsers.
* [https://beatdetektor.svn.sourceforge.net/svnroot/beatdetektor/trunk/core/js/beatdetektor.js Beat Detektor] by Charles Cliffe, uses dsp.js to add beat detection.
* [https://beatdetektor.svn.sourceforge.net/svnroot/beatdetektor/trunk/core/js/beatdetektor.js Beat Detektor] by Charles Cliffe, uses dsp.js to add beat detection.
* [https://github.com/jussi-kalliokoski/audiolib.js/ audiolib.js] by Jussi Kalliokoski, a powerful audio tools library for JavaScript, compatible with the Audio Data API and Chrome's Audio API.
* [https://github.com/jussi-kalliokoski/audiolib.js audiolib.js] by Jussi Kalliokoski, a powerful audio tools library for JavaScript, compatible with the Audio Data API and Chrome's Audio API.
* [https://github.com/oampo/Audiolet Audiolet] - A JavaScript library for real-time audio synthesis and composition from within the browser
* [https://github.com/grantgalitz/XAudioJS XAudioJS] - A JavaScript library that provides a raw audio sample writing access to the mozilla audio data and web audio APIs. Provides a basic write and callback system so the developer can be assured to have gapless audio for these two APIs. Also provides a fallback WAV PCM data URI generator that is not guaranteed to be gapless.
* [http://www.gregjopa.com/2011/05/calculate-note-frequencies-in-javascript-with-music-js/ Music.js, library containing functions and data sets to generate notes, intervals, chords, and scales]
* [https://github.com/BillyWM/jsmodplayer Javascript .MOD and .XM music player]


=== Working Audio Data Demos ===
=== Working Audio Data Demos ===
'''NOTE: we recently took down two servers that were hosting many of these demos.  We are working to find a new home for them.'''


A number of working demos have been created, including:
A number of working demos have been created, including:
Line 553: Line 569:


* Audio Visualizations
* Audio Visualizations
** http://tllabs.io/audiopaper/ paper.js audio visualization
** http://traction.untergrund.net/slamdown/
** http://www.nihilogic.dk/labs/pocket_full_of_html5/ (Demo by Jacob Seidelin)
** http://www.nihilogic.dk/labs/pocket_full_of_html5/ (Demo by Jacob Seidelin)
** http://weare.buildingsky.net/processing/dsp.js/examples/fft.html
** http://weare.buildingsky.net/processing/dsp.js/examples/fft.html
** http://www.storiesinflight.com/jsfft/visualizer/index.html (Demo by Thomas Sturm)
** http://www.storiesinflight.com/jsfft/visualizer/index.html (Demo by Thomas Sturm)
** http://www.grantgalitz.org/sound_test/ WAV Decoder & Visualizer (Pre-loaded)
** http://www.grantgalitz.org/wav_player/ WAV Decoder & Visualizer (Load in your own .wav)


* Applying Realtime Audio Effects
* Applying Realtime Audio Effects
** Volume, pitch, etc. UI for audio - https://developer.mozilla.org/en-US/demos/detail/voron (homepage: http://kievII.net)
** JS IIR Filter http://weare.buildingsky.net/processing/dsp.js/examples/filter.html (video [http://vimeo.com/11335434 here])  
** JS IIR Filter http://weare.buildingsky.net/processing/dsp.js/examples/filter.html (video [http://vimeo.com/11335434 here])  
** Vocodes a formant with a carrier wave http://weare.buildingsky.net/processing/dsp.js/examples/vocoder.html
** Vocodes a formant with a carrier wave http://weare.buildingsky.net/processing/dsp.js/examples/vocoder.html
Line 566: Line 587:


* Generating and Playing Audio
* Generating and Playing Audio
** http://bitterspring.net/blog/2012/01/25/morning-star-synth-0-1-released/
** http://onlinetonegenerator.com/
** [http://jsmad.org/ mp3 decoder in js]
** [http://cosinusoidally.github.com/mp2dec.js/ mp2 decoder in js]
** [http://www.oampo.co.uk/2011/05/technocrat/ Ambient techno machine]
** [http://www.gregjopa.com/2011/05/calculate-note-frequencies-in-javascript-with-music-js/ Music.js, library containing functions and data sets to generate notes, intervals, chords, and scales]
** [https://hacks.mozilla.org/2011/01/html5guitar/ HTML5 Guitar Tab Player]
** [https://hacks.mozilla.org/2011/01/html5guitar/ HTML5 Guitar Tab Player]
** [http://audioscene.org/scene-files/humph/sfxr/ sfxr.js] - sound effect generator/editor for video games.
** [http://automata.cc/src/vivace/experiments/matrix.html Tone matrix using Audiolet.js]
** [http://www.oampo.co.uk/labs/audiolet-demo/ Generating music in JS via audiolet.js], [http://www.oampo.co.uk/labs/breakbeat/ breakbeat demo]
** [http://humphd.github.com/sfxr.js/ sfxr.js] - sound effect generator/editor for video games.
** [http://jonbro.tk/blog/2010/09/19/html_5_chip_tracker.html JavaScript Chip Tracker app] (demo by Jonathan Brodsky)
** [http://jonbro.tk/blog/2010/09/19/html_5_chip_tracker.html JavaScript Chip Tracker app] (demo by Jonathan Brodsky)
** JavaScript Audio Sampler http://weare.buildingsky.net/processing/dsp.js/examples/sampler.html
** JavaScript Audio Sampler http://weare.buildingsky.net/processing/dsp.js/examples/sampler.html
Line 574: Line 603:
** Random Noise Generation http://weare.buildingsky.net/processing/dsp.js/examples/nowave.html
** Random Noise Generation http://weare.buildingsky.net/processing/dsp.js/examples/nowave.html
** JS Multi-Oscillator Synthesizer http://weare.buildingsky.net/processing/dsp.js/examples/synthesizer.html (video [http://vimeo.com/11411533 here])
** JS Multi-Oscillator Synthesizer http://weare.buildingsky.net/processing/dsp.js/examples/synthesizer.html (video [http://vimeo.com/11411533 here])
** Bloop http://audioscene.org/scene-files/yury/examples/bloop-ea/bloop-audiodata.html
** Bloop http://async5.org/audiodata/examples/bloop-ea/bloop-audiodata.html
** JavaScript Text to Speech engine http://audioscene.org/scene-files/yury/tts/index.html
** JavaScript Text to Speech engine http://async5.org/audiodata/tts/index.html
** Toy Piano http://audioscene.org/scene-files/yury/examples/piano.html (and the sample-based piano http://audioscene.org/scene-files/yury/examples/piano-s/piano2.html)
** Toy Piano http://async5.org/audiodata/examples/piano.html (and the sample-based piano http://async5.org/audiodata/examples/piano-s/piano2.html)
** Csound Shaker Instrument http://audioscene.org/scene-files/yury/csound/shaker.htm and Bar Instrument http://audioscene.org/scene-files/yury/csound/bar.htm
** Csound Shaker Instrument http://async5.org/audiodata/csound/shaker.htm and Bar Instrument http://async5.org/audiodata/csound/bar.htm
** Canon Theremin Piano http://mtg.upf.edu/static/media/canon-theremin-piano.html (by Zacharias Vamvakousis zackbam@gmail.com).
** Canon Theremin Piano http://mtg.upf.edu/static/media/canon-theremin-piano.html (by Zacharias Vamvakousis zackbam@gmail.com).
** Manipulate music example using mouse and accelerometer http://blog.dt.in.th/stuff/audiodata/ (Thai Pangsakulyanont)
** Manipulate music example using mouse and accelerometer http://blog.dt.in.th/stuff/audiodata/ (Thai Pangsakulyanont)
Line 583: Line 612:
** Modular Synthesizer with MIDI, control and audio ports. http://www.niiden.com/jstmodular/ (Jussi Kalliokoski)
** Modular Synthesizer with MIDI, control and audio ports. http://www.niiden.com/jstmodular/ (Jussi Kalliokoski)
** Dual-axis Theremin controlling pitch and volume with cursor position. http://stu.ie/?p=2599 (Stuart Gilbert)
** Dual-axis Theremin controlling pitch and volume with cursor position. http://stu.ie/?p=2599 (Stuart Gilbert)
** JavaScript "Image to Sound" generator http://bp.bai-hua.org/_/audio_test6.html (ZhangJW)
** JavaScript "Image to Sound" generator http://zhangjw.bai-hua.org/audio_test6.html (ZhangJW)
** XAudioJS library test page http://www.grantgalitz.org/sound_test/


* Beat Detection (also showing use of WebGL for 3D visualizations)
* Beat Detection (also showing use of WebGL for 3D visualizations)
** http://people.mozilla.com/~prouget/demos/boomboom/index.html
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor1HD.html (video [http://vimeo.com/11345262 here])
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor1HD.html (video [http://vimeo.com/11345262 here])
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor2HD.html (video of older version [http://vimeo.com/11345685 here])
** http://cubicvr.org/CubicVR.js/bd3/BeatDetektor2HD.html (video of older version [http://vimeo.com/11345685 here])
Line 599: Line 630:
** API Example: [http://audioscene.org/?p=279 Ambient Extraction Mixer]
** API Example: [http://audioscene.org/?p=279 Ambient Extraction Mixer]
** API Example: [http://audioscene.org/?p=302 Worker Thread Audio Processing]
** API Example: [http://audioscene.org/?p=302 Worker Thread Audio Processing]
* Audio Games
** http://www.oampo.co.uk/labs/fireflies/
** http://www.oampo.co.uk/labs/siren-song/


=== Demos Needing to be Updated to New API ===
=== Demos Needing to be Updated to New API ===
Teoli2003
Confirmed users
504

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/285936...657449"

Navigation menu