Gecko:MediaRecorder

Media recoding architecture

Errata

• Vorbos => Theora
• We can not use VideoFrameContainer/AudioSystem directly, use adapter instead

MediaRecorder

Reference spec

https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/RecordingProposal.html

Related bug

bug 803414 Audio Recording - Web API & Implementation
bug 834165 Implement BlobEvent
bug 825110 [meta] Porting WebRTC video_module for B2G
bug 825112 [meta] Porting WebRTC audio_module for B2G
media encoder
audio encoder
video encoder
media writer
ogg writer

MediaEncoder Draft

API draft

/**
 * Implement a queue-like interface between MediaStream and encoder
 * since encoded may take long time to process one segment while new 
 * segment comes.
 */
class MediaSegmentAdapter {
public:
   enum BufferType {
       AUDIO,
       VIDEO
   }

   /* AppendSegment may run on MediaStreamGraph thread to avoid race condition */
   void AppendSegment(MediaSegment) = 0;
   void SetBufferLength(size_t);
   size_t GetBufferLength();

   MediaSegment DequeueSegment();

protected:
   MediaSegmentAdapter();
   Queue<MediaSegment> mBuffer;

   friend class Encoder;
}

/**
 * A base class for video type adapters, which provide basic implementation 
 * e.g, copy the frame
 */
class VideoAdapter : MediaSegmentAdapter {
public:
   /* This version deep copy/color convert the input buffer into a local buffer */
   void AppendSegment(MediaSegment);
}

/**
 * In FirefoxOS, we have hardware encoder and camera output platform-specific 
 * buffer which may give better performance
 */
class GrallocAdapter : MediaSegmentAdapter {
public: 
   /* This version |do not| copy frame data, but queue the GraphicBuffer directly
      which can be used with SurfaceMediaSource or other SurfaceFlinger compatible 
      mechanism for hardware supported video encoding */
   void AppendSegment(MediaSegment);
}

/**
 * Similar to VideoAdapter, and since audio codecs may need |collect| enough data
 * then real encode it, we may implement raw buffer with some specific length and
 * collect data into the buffer 
 */
class AudioAdapter : MediaSegmentAdapter {
public: 
   /* Copy/resample the data into local buffer */
   void AppendSegment(MediaSegment);
}

/**
 * This class is similar to MediaStreamGraph.
 * It take response to initialize all the component and link them together.
 */
class MediaEncoder : MediaStreamListener {
public:
   /* Callback functions for MediaStream */
   void NotifyConsumption(MediaStreamGraph, Consumption);
   void NotifyPull(MediaStreamGraph, StreamTime);
   void NotifyBlockingChanged(MediaStreamGraph, Blocking);
   void NotifyHasCurrentData(MediaStreamGraph, bool);
   void NotifyOutput(MediaStreamGraph);
   void NotifyFinished(MediaStreamGraph);
   /* Queue the MediaSegment into correspond adapters */
   /* XXX: Is it possible to determine Audio related paramenters from this callback? 
           Or we have to query them from MediaStream directly? */
   /* AppendSegment into Adapter need block MediaStreamGraph thread to avoid race condition
      and we should schedule one encoding loop if any track updated */
   void NotifyQueuedTrackChanges(MediaStreamGraph, TrackID, TrackRate, TrackTicks, uint32_t, MediaSegment);

   /* Callback functions to JS */
   void SetDataAvailableCallback()
   void SetErrorCallback()
   void SetMuteTrackCallback()  /*NOT IMPL*/
   void SetPauseCallback()
   void SetPhotoCallback()  /*NOT IMPL*/
   void SetRecordingCallback()
   void SetResumeCallback()
   void SetStopCallback()
   void SetUnmuteTrackCallback()  /*NOT IMPL*/
   void SetWarningCallback()

   /* Option query functions */
   nsArray<String> GetSupportMIMEType()
   Pair<int, int> GetSupportWidth() 
   Pair<int, int> GetSupportHeight()

   /* Set requested encoder */
   void SetMIMEType();
   void SetVideoWidth();
   void SetVideoHeight();

   /* JS control functions */
   void MuteTrack(TrackID)  /*NOT IMPL*/
   void Pause()
   void Record()
   void RequestData  /*NOT IMPL*/
   void Resume()
   void SetOptions()
   void Stop()
   void TakePhoto  /*NOT IMPL*/
   void UnmuteTrack  /*NOT IMPL*/

   /* initial internal state and codecs */
   void Init()

   /* create MediaEncoder for given MediaStream */
   MediaEncoder(MediaStream);

private: 
   void QueueVideoSegments();
   void QueueAudioSegments();
   void SelectCodec();
   void ConfigCodec();
   void SetVideoQueueSize();
   void SetAudioQueueSize();

   /* data member */
   MediaSegmentAdapter mVideoSegments;
   MediaSegmentAdapter mAudioSegments;
   Encoder mVideoEncoder;
   Encoder mAudioEncoder;
   MediaEncoder mMediaEncoder;
   Thread mEncoderThread;
}

/**
 * Different codecs usually support some codec specific parameters which
 * we may take advantage of.
 *
 * Let each implementation provide its own parameter set, and use common
 * params if no special params requested.
 */
union CodecParams {
   OpusParams opusParams;
   TheoraParams theoraParams;
   MPEG4Params mpeg4Params;
   // etc.
}

/**
 * base class for general codecs:
 *
 * we generally do not implement codec ourself, but we need a generic interface
 * to capsulate it.
 *
 * For example, if we want to support opus, we should create a OpusCodec and let
 * it inherit this base class(by inherit AudioCodec), and implement OpusCodec by 
 * utilize libopus API.
 */
class Encoder {
public:
   enum EncodingState {
       COLLOCTING, /* indicate the encoder still wait enough data to be encoded */
       ENCODING,   /* there is enough data to be encoded, but incomplete */
       ENCODED,    /* indicate there is some output can be get from this codec */
   }

   Encoder();
   nsresult Init() = 0;

   /* Let Encoder setup buffer length based on codec characteristic
      e.g. Stagefright video codecs may only use 1 buffer since the buffer maybe shared between hardwares */
    

   /* Mimic android::CameraParameter to collect backend codec related params in general class */
   CodecParams GetParams() = 0;
   nsresult SetParams(CodecParams) = 0;
              
   /* Start the encoder, if the encoder got its own thread, create the thread here */
   nsresult Encode() = 0;

   /* Read the encoded data from encoder, check the status before attempt to read, otherwise error would returned */
   EncoderState GetCurrentState();
   nsresult GetEncodedData(MediaSegment& encData) = 0;

   /* codec specific header to describe self type/version/etc. */
   Metadata GetCodecHeader();

   /* force the encoder to output current available data */
   /* XXX: this maybe required to support MediaEncoder::Request, but may not supported by all encoder backend */
   void Flush() = 0;

private:
   MediaSegmentAdapter mQueue;
}

class AudioTrackEncoder : public Encoder {
public:
   /* AudioCodec may need collect enough buffers to be encode, return COLLECT as needed */
   EncoderStatus Encode(MediaBuffer in, void* out, size_t length) = 0;

private:
   bool IsDataEnough();
   void* mLocalBuffer[MIN_FRAMES];
}

class VideoTrackEncoder : public Encoder {
public: 
   EncoderStatus Encode(MediaBuffer in, void* out, size_t length) = 0;
}

class OpusEncoder : public AudioTrackEncoder {
   // Use libopus to encode audio
private:
   // libopus ctx, etc...
}

class TheoraEncoder : public VideoTrackEncoder {
   // Use libtheora to encode video
private:
   // libtheora encoder is not blocking, thus we have to loop until frame complete
}

/**
 * Generic base class for container writer
 *
 * Similar to MediaCodec and we separate container and codec for future extension.
 */
class MediaWriter {
public:
   void AddTrack(Encoder);

   /* Block until container packet write done*/
   nsTArray<char> GetPacket(); 
} 

class OggWriter {
   // Use libogg to write container
protected:
   // libogg context and others
   VideoTrackEncoder mVideoEncoder; // e.g. TheoraEncoder
   AudioTrackEncoder mAudioEncoder; // e.g. OpusEncoder/VorbisEncoder
}

MediaRecorder

 [Constructor (MediaStream stream)]
 interface MediaRecorder : EventTarget  {
   readonly attribute MediaStream    stream;
   readonly attribute RecordingState state;
            attribute EventHandler   onrecording;
            attribute EventHandler   onstop;
            attribute EventHandler   ondataavailable;
            attribute EventHandler   onpause;
            attribute EventHandler   onresume;
             attribute EventHandler   onmutetrack;   /*NOT IMPL*/  
            attribute EventHandler   onunmutetrack; /*NOT IMPL*/  
            attribute EventHandler   onphoto;       /*NOT IMPL*/  
            attribute EventHandler   onerror;
            attribute EventHandler   onwarning;     /*NOT IMPL*/  
   void                              record (optional long timeSlice);
   void                              stop ();
   void                              pause();
   void                              resume();
   void                              muteTrack(DOMString trackID);    /*NOT IMPL*/ 
   void                              unmuteTrack(DOMString trackID);  /*NOT IMPL*/ 
   void                              takePhoto(DOMString trackID);    /*NOT IMPL*/ 
   void                              requestData();
   AvailableRecordingFormats         getOptions (); /*NOT IMPL*/  
   void                              setOptions (RecordingFormat RequestedFormats);  /*NOT IMPL*/  
 };

 enum RecordingState {
   "inactive",
   "recording",
   "paused",
 };

 interface RecordingError {
   readonly attribute DOMString  name;
   readonly attribute DOMString? message;
 };

 enum RecordingErrorName {
   "OUT_OF_MEMORY",
   "ILLEGAL_STREAM_MODIFICATION"
   "OTHER_RECORDING_ERROR"
 };

Working flow

1. MediaEncoder create MediaCodecs, MediaWriter based on request
   1. MediaEncoder create MediaSegmentAdapters from MediaCodecs
   2. MediaEncoder add MediaCodecs into MediaWriter tracks
2. MediaEncoder register callback to MediaStream
3. MediaStreamGraph thread callback to MediaEncoder when stream update
4. MediaEncoder queue new MediaSegments into each MediaSegmentAdapters based on its type
   1. MediaSegmentAdapter copy/enqueue/color convert/etc... the data and queue them up
5. MediaEncoder post a task to encoder thread
6. Encoder thread ask MediaWriter for Packet
   1. If MediaWriter::GetPacket called for first time
      1. get Codec specific headers first, and produce Write header/metadata packet
   2. Otherwise
      1. MediaWriter ask Codecs for encoded data
      2. MediaWriter write packets
7. MediaEncoder call onRecordingCallback with raw data or nsArray
8. MediaRecord API post encoded data blob to Javascript layer

NOTE: step 8 are not specified in this API

TEST CASE

• Recording media stream audio data, can be played by audio tag
• MediaRecorder state machine check

Problems

• General codecs do not describe metadata
  • Codec type information have to be write done by some other mechenism
• Some codecs collect enough data to produce output, MediaCodec::GetEncodedData is not adequate.
  • Writer should query MediaCodec state before attempt to read data
  • Some container force stream interleave, we may need some sync mechanism
• Since encoder may pending, EOS event may need some extra handling

=> Messaging related detail should be determinated until real implementation

Notes

• We will only implement Audio related part in current stage
• Some interaction between MediaEncoder and MediaRecorder is indeterminated, the affected function will not implemented at this stage (marked with /*NOT IMPL*/)

References

libogg/libopus/libtheora: http://www.xiph.org/ogg/