Gecko:MediaRecorder

Media recoding architecture

Errata

• Vorbos => Theora
• We can not use VideoFrameContainer/AudioSystem directly, use adapter instead

MediaRecorder

Reference spec

https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/RecordingProposal.html

Related bug

bug 803414 Audio Recording - Web API & Implementation
bug 834165 Implement BlobEvent
bug 825110 [meta] Porting WebRTC video_module for B2G
bug 825112 [meta] Porting WebRTC audio_module for B2G
media encoder
audio encoder
video encoder
media writer
ogg writer

MediaEncoder Draft

API draft

class MediaSegmentAdapter {
public:
   enum BufferType {
       AUDIO,
       VIDEO
   }

   /* AppendSegment may run on MediaStreamGraph thread to avoid race condition */
   void AppendSegment(MediaSegment) = 0;
   void SetBufferLength(size_t);
   size_t GetBufferLength();

   MediaSegment DequeueSegment();

protected:
   MediaSegmentAdapter();
   Queue<MediaSegment> mBuffer;

   friend class MediaCodec;
}

class VideoAdapter : MediaSegmentAdapter {
public:
   /* This version deep copy/color convert the input buffer into a local buffer */
   void AppendSegment(MediaSegment);
}

class GrallocAdapter : MediaSegmentAdapter {
public: 
   /* This version |do not| copy frame data, but queue the GraphicBuffer directly
      which can be used with SurfaceMediaSource or other SurfaceFlinger compatible 
      mechanism for hardware supported video encoding */
   void AppendSegment(MediaSegment);
}

class AudioAdapter : MediaSegmentAdapter {
public: 
   /* Copy/resample the data into local buffer */
   void AppendSegment(MediaSegment);
}

class MediaEncoder : MediaStreamListener {
public:
   /* Callback functions for MediaStream */
   void NotifyConsumption(MediaStreamGraph, Consumption);
   void NotifyPull(MediaStreamGraph, StreamTime);
   void NotifyBlockingChanged(MediaStreamGraph, Blocking);
   void NotifyHasCurrentData(MediaStreamGraph, bool);
   void NotifyOutput(MediaStreamGraph);
   void NotifyFinished(MediaStreamGraph);
   /* Queue the MediaSegment into correspond adapters */
   /* XXX: Is it possible to determine Audio related paramenters from this callback? 
           Or we have to query them from MediaStream directly? */
   /* AppendSegment into Adapter need block MediaStreamGraph thread to avoid race condition
      and we should schedule one encoding loop if any track updated */
   void NotifyQueuedTrackChanges(MediaStreamGraph, TrackID, TrackRate, TrackTicks, uint32_t, MediaSegment);

   /* Callback functions to JS */
   void SetDataAvailableCallback()
   void SetErrorCallback()
   void SetMuteTrackCallback()  /*NOT IMPL*/
   void SetPauseCallback()
   void SetPhotoCallback()  /*NOT IMPL*/
   void SetRecordingCallback()
   void SetResumeCallback()
   void SetStopCallback()
   void SetUnmuteTrackCallback()  /*NOT IMPL*/
   void SetWarningCallback()

   /* Option query functions */
   nsArray<String> GetSupportMIMEType()
   Pair<int, int> GetSupportWidth() 
   Pair<int, int> GetSupportHeight()

   /* Set requested encoder */
   void SetMIMEType();
   void SetVideoWidth();
   void SetVideoHeight();

   /* JS control functions */
   void MuteTrack(TrackID)  /*NOT IMPL*/
   void Pause()
   void Record()
   void RequestData  /*NOT IMPL*/
   void Resume()
   void SetOptions()
   void Stop()
   void TakePhoto  /*NOT IMPL*/
   void UnmuteTrack  /*NOT IMPL*/

   /* initial internal state and codecs */
   void Init()

   /* create MediaEncoder for given MediaStream */
   MediaEncoder(MediaStream);

private: 
   void QueueVideoSegments();
   void QueueAudioSegments();
   void SelectCodec();
   void ConfigCodec();
   void SetVideoQueueSize();
   void SetAudioQueueSize();

   /* data member */
   MediaSegmentAdapter mVideoSegments;
   MediaSegmentAdapter mAudioSegments;
   MediaCodec mVideoCodec;
   MediaCodec mAudioCodec;
   MediaWriter mWriter;
   Thread mEncoderThread;
}

class CodecParams {
   /* key value pair for detailed codec settings */
   Map<Key, Value> mMap;
}

/* base class for general codecs */
class MediaCodec {
public:
   enum EncodingState {
       COLLOCTING, /* indicate the encoder still wait enough data to be encoded */
       ENCODING,   /* there is enough data to be encoded, but incomplete */
       ENCODED,    /* indicate there is some output can be get from this codec */
   }

   MediaCodec();
   status_t Init() = 0;

   /* Let MediaCodec setup buffer length based on codec characteristic
      e.g. Stagefright video codecs may only use 1 buffer since the buffer maybe shared between hardwares */
    

   /* Mimic android::CameraParameter to collect backend codec related params in general class */
   CodecParams GetParams() = 0;
   status_t SetParams(CodecParams) = 0;
              
   /* Start the encoder, if the encoder got its own thread, create the thread here */
   status_t Encode() = 0;

   /* Read the encoded data from encoder, check the status before attempt to read, otherwise error would returned */
   EncoderState GetCurrentState();
   status_t GetEncodedData(MediaSegment& encData) = 0;

   /* codec specific header to describe self type/version/etc. */
   Metadata GetCodecHeader();

   /* force the encoder to output current available data */
   /* XXX: this maybe required to support MediaEncoder::Request, but may not supported by all encoder backend */
   void Flush() = 0;

private:
   MediaSegmentAdapter mQueue;
}

class AudioCodec : public MediaCodec {
public:
   /* AudioCodec may need collect enough buffers to be encode, return COLLECT as needed */
   EncoderStatus Encode(MediaBuffer in, void* out, size_t length) = 0;

private:
   bool IsDataEnough();
   void* mLocalBuffer[MIN_FRAMES];
}

class VideoCodec : public MediaCodec {
public: 
   EncoderStatus Encode(MediaBuffer in, void* out, size_t length) = 0;
}

class OpusCodec : public AudioCodec {
   // Use libopus to encode audio
private:
   // libopus ctx, etc...
}

class TheoraCodec : public VideoCodec {
   // Use libtheora to encode video
private:
   // libtheora encoder is not blocking, thus we have to loop until frame complete
}

class MediaWriter {
public:
   void AddTrack(MediaCodec);

   /* Block until container packet write done*/
   nsTArray<char> GetPacket(); 
} 

class OggWriter {
   // Use libogg to write container
public:
   // libogg context and others
}

MediaRecorder

 [Constructor (MediaStream stream)]
 interface MediaRecorder : EventTarget  {
   readonly attribute MediaStream    stream;
   readonly attribute RecordingState state;
            attribute EventHandler   onrecording;
            attribute EventHandler   onstop;
            attribute EventHandler   ondataavailable;
            attribute EventHandler   onpause;
            attribute EventHandler   onresume;
             attribute EventHandler   onmutetrack;   /*NOT IMPL*/  
            attribute EventHandler   onunmutetrack; /*NOT IMPL*/  
            attribute EventHandler   onphoto;       /*NOT IMPL*/  
            attribute EventHandler   onerror;
            attribute EventHandler   onwarning;     /*NOT IMPL*/  
   void                              record (optional long timeSlice);
   void                              stop ();
   void                              pause();
   void                              resume();
   void                              muteTrack(DOMString trackID);    /*NOT IMPL*/ 
   void                              unmuteTrack(DOMString trackID);  /*NOT IMPL*/ 
   void                              takePhoto(DOMString trackID);    /*NOT IMPL*/ 
   void                              requestData();
   AvailableRecordingFormats         getOptions (); /*NOT IMPL*/  
   void                              setOptions (RecordingFormat RequestedFormats);  /*NOT IMPL*/  
 };

 enum RecordingState {
   "inactive",
   "recording",
   "paused",
 };

 interface RecordingError {
   readonly attribute DOMString  name;
   readonly attribute DOMString? message;
 };

 enum RecordingErrorName {
   "OUT_OF_MEMORY",
   "ILLEGAL_STREAM_MODIFICATION"
   "OTHER_RECORDING_ERROR"
 };

Working flow

1. MediaEncoder create MediaCodecs, MediaWriter based on request
   1. MediaEncoder create MediaSegmentAdapters from MediaCodecs
   2. MediaEncoder add MediaCodecs into MediaWriter tracks
2. MediaEncoder register callback to MediaStream
3. MediaStreamGraph thread callback to MediaEncoder when stream update
4. MediaEncoder queue new MediaSegments into each MediaSegmentAdapters based on its type
   1. MediaSegmentAdapter copy/enqueue/color convert/etc... the data and queue them up
5. MediaEncoder post a task to encoder thread
6. Encoder thread ask MediaWriter for Packet
   1. If MediaWriter::GetPacket called for first time
      1. get Codec specific headers first, and produce Write header/metadata packet
   2. Otherwise
      1. MediaWriter ask Codecs for encoded data
      2. MediaWriter write packets
7. MediaEncoder call onRecordingCallback with raw data or nsArray
8. MediaRecord API post encoded data blob to Javascript layer

NOTE: step 8 are not specified in this API

Problems

• General codecs do not describe metadata
  • Codec type information have to be write done by some other mechenism
• Some codecs collect enough data to produce output, MediaCodec::GetEncodedData is not adequate.
  • Writer should query MediaCodec state before attempt to read data
  • Some container force stream interleave, we may need some sync mechanism
• Since encoder may pending, EOS event may need some extra handling

Notes

• We will only implement Audio related part in current stage
• Some interaction between MediaEncoder and MediaRecorder is indeterminated, the affected function will not implemented at this stage (marked with /*NOT IMPL*/)

References

libogg/libopus/libtheora: http://www.xiph.org/ogg/