Platform/Features/WebRTC

Please use "Edit with form" above to edit this page.

Status

WebRTC (formerly Video (and Audio) Conferencing)
Stage	Planning
Status	In progress
Release target	Phase 1 - TBD (first release planned this year, 2012)
Health	OK
Status note	We have moved forward with development, even though the spec is still being hashed out. Our initial release may be prefixed.

{{#set:Feature name=WebRTC (formerly Video (and Audio) Conferencing)

|Feature stage=Planning |Feature status=In progress |Feature version=Phase 1 - TBD (first release planned this year, 2012) |Feature health=OK |Feature status note=We have moved forward with development, even though the spec is still being hashed out. Our initial release may be prefixed. }}

Team

Product manager	Maire Reavy
Directly Responsible Individual	Maire Reavy
Lead engineer	Randell Jesup
Security lead	Curtis Koenig (Maire to verify)
Privacy lead	Sid Stamm (Maire to verify)
Localization lead	`
Accessibility lead	TBD
QA lead	Jason Smith
UX lead	Jennifer Morrow
Product marketing lead	TBD
Operations lead	`
Additional members	Cisco Team (Enda Mannion, Suhas Nandakumar, Ethan Hunt), Eric Rescorla, Rob O'Callahan (MediaStreams piece), Ralph Giles, TIm Terriberry, Anant Naranyanan (part time, mostly spec work after IETF 83)

{{#set:Feature product manager=Maire Reavy

|Feature feature manager=Maire Reavy |Feature lead engineer=Randell Jesup |Feature security lead=Curtis Koenig (Maire to verify) |Feature privacy lead=Sid Stamm (Maire to verify) |Feature localization lead=` |Feature accessibility lead=TBD |Feature qa lead=Jason Smith |Feature ux lead=Jennifer Morrow |Feature product marketing lead=TBD |Feature operations lead=` |Feature additional members=Cisco Team (Enda Mannion, Suhas Nandakumar, Ethan Hunt), Eric Rescorla, Rob O'Callahan (MediaStreams piece), Ralph Giles, TIm Terriberry, Anant Naranyanan (part time, mostly spec work after IETF 83) }}

Open issues/risks

• VP8 is the only open, modern video codec under consideration. There is heavy debate going on in the IETF working group about what video codec should be mandated by the spec. Some are arguing that H.264 should be selected because they prefer the patent licensing situation there (MPEG-LA) and more mature technology. Without a common video codec, the spec could fail to gain acceptance and adoption.
• Overall system latency (delay) might be an issue; if it is architectural changes may be needed. Testing will need to be done.
• Privacy UI/UX for this feature is a challenge since this involves camera and microphone data. Security will be difficult due to the wish for applications to be able to provide useful interfaces, in conflict with the need to guarantee users are safe with regards to access to their camera and microphones. This design is being discussed with the security team.
• Good congestion control is essential for real-time video/audio communication. The algorithm for this is still being developed. We can’t make our first release until we have a good algorithm implemented in the Firefox code.

Stage 1: Definition

1. Feature overview

This feature will allow people using the web platform to include video/audio conferencing and associated data as part of their websites and applications. The main attributes include:

• Using the web as the setup end point for a "call." This is different than other calling systems, which use SIP, XMPP or other signaling systems.
• Allowing sites access to video camera feed for other purposes than a call, such as graphically manipulating the video feed or allowing users to record video
• Once capabilities are exchanged, a point-to-point connection is established between browsers when possible. This includes NAT traversal.
• Video is displayed as a primary object in the browser and can be part of content.
• Data can be exchanged along with the video and audio.

Phase 1

We will support applications that do the following:

• Call a friend using a social API like Facebook. From the chat/online window the user can click on a person and start a video call with them in the browser. The video/audio/data should be securely transmitted.
• Video conferencing / sharing slides from inside an app. The user calls someone from inside an app and is able to share files, images, slides, etc through the P2P connection as he/she is talking via the data channel.
• Games that integrate video, audio and data sharing. For example a chess game where you see the video and hear the audio of your opponent and play the game live using WebRTC’s data channels.

Note: Basic Interoperation between Firefox's WebRTC code and Chrome's WebRTC code will be supported as part of phase 1. This should include video, audio, and data channels interop if Chrome supports data channels in time.

The following is a nice-to-have use case that we may or may not support in phase 1. If we don't support it in phase 1, we will support it in phase 2:

• Remote assistance/co-pilot - you "look over the shoulder" of someone as they browse (optionally controlling the mouse/keyboard), while also talking with them over audio or audio+video.

Phase 2

• Support for Android Mobile devices.
• Probably support for B2G Mobile devices (TBD)

For reference only:

• IETF Use Case Document

3. Dependencies

• Integration of the newly released WebRTC code.
• Work product from IETF and W3 working groups
• Design of getUserMedia from Media Task Force
• Integration of Rob O'Callahan's Media API work.

4. Requirements

Must Have:

• Security design review
• Security implementation review

• The ability to make a call from a web page using a machine's built-in mic and camera.
• Should be able to work with most (80-90%) of NAT setups across all geographies. Note that certain configurations will require a relay.
• Support for royalty-free voice-friendly audio codecs. Opus strongly preferred.
• Support for royalty-free video codecs
• Audio/Video/Data must be secure (note that no security is absolute)
• A Web API that's been sent to the public working groups for feedback.
• A Web API and transport for data that accompanies the audio and video.
• Congestion control (Automatic bandwidth adjustment) required to be in and debugged prior to Firefox's WebRTC code's fire release
• Ability to add/remove video from an active call and prior to a call being placed or received

Nice to Have:

• Video resolution switching
• If hardware support exists for encoding & decoding VP8, support for video on Mobile.

Non-goals

We are not trying to support anything other than the items mentioned in this document. This page is primarily focused on what's required to ship the first phase of the feature.

Stage 2: Design

5. Functional specification

• Audio codecs planned: G.711 & Opus. (Although royalty/license-free, we have no plans to support iLBC, iSAC and G.722)
• Video codecs planned: VP8

API Docs:

• Robert's MediaStream Processing API Proposal

6. User experience design

Latest spec can be viewed here: people.mozilla.com/~jboriss/specs/webrtc_spec.png

Stage 3: Planning

7. Implementation plan

We created our first public demo at IETF 83 (Paris). We were able to demo video/audio calls and working data channels. We need to add signaling (JSEP) support and PeerConnection support. We’re creating a project planning map of work to be done and we have a WebRTC tracking bug, bug 665909

8. Reviews

Security review

It's important to realize that this is the first step in a plan to bring video to the platform. From both a security and privacy standpoint this means we'll need to be reviewing it with an eye towards an eye towards future use cases as well as what's scoped in this one page.

Privacy review

`

Localization review

`

Accessibility

`

Quality Assurance review

`

Operations review

`

Stage 4: Development

9. Implementation

`

Stage 5: Release

10. Landing criteria

` {{#set:Feature open issues and risks=* VP8 is the only open, modern video codec under consideration. There is heavy debate going on in the IETF working group about what video codec should be mandated by the spec. Some are arguing that H.264 should be selected because they prefer the patent licensing situation there (MPEG-LA) and more mature technology. Without a common video codec, the spec could fail to gain acceptance and adoption.

• Overall system latency (delay) might be an issue; if it is architectural changes may be needed. Testing will need to be done.
• Privacy UI/UX for this feature is a challenge since this involves camera and microphone data. Security will be difficult due to the wish for applications to be able to provide useful interfaces, in conflict with the need to guarantee users are safe with regards to access to their camera and microphones. This design is being discussed with the security team.
• Good congestion control is essential for real-time video/audio communication. The algorithm for this is still being developed. We can’t make our first release until we have a good algorithm implemented in the Firefox code.

|Feature overview=This feature will allow people using the web platform to include video/audio conferencing and associated data as part of their websites and applications. The main attributes include:

• Using the web as the setup end point for a "call." This is different than other calling systems, which use SIP, XMPP or other signaling systems.
• Allowing sites access to video camera feed for other purposes than a call, such as graphically manipulating the video feed or allowing users to record video
• Once capabilities are exchanged, a point-to-point connection is established between browsers when possible. This includes NAT traversal.
• Video is displayed as a primary object in the browser and can be part of content.
• Data can be exchanged along with the video and audio.

|Feature users and use cases===Phase 1== We will support applications that do the following:

• Call a friend using a social API like Facebook. From the chat/online window the user can click on a person and start a video call with them in the browser. The video/audio/data should be securely transmitted.
• Video conferencing / sharing slides from inside an app. The user calls someone from inside an app and is able to share files, images, slides, etc through the P2P connection as he/she is talking via the data channel.
• Games that integrate video, audio and data sharing. For example a chess game where you see the video and hear the audio of your opponent and play the game live using WebRTC’s data channels.

Note: Basic Interoperation between Firefox's WebRTC code and Chrome's WebRTC code will be supported as part of phase 1. This should include video, audio, and data channels interop if Chrome supports data channels in time.

The following is a nice-to-have use case that we may or may not support in phase 1. If we don't support it in phase 1, we will support it in phase 2:

• Remote assistance/co-pilot - you "look over the shoulder" of someone as they browse (optionally controlling the mouse/keyboard), while also talking with them over audio or audio+video.

Phase 2

• Support for Android Mobile devices.
• Probably support for B2G Mobile devices (TBD)

For reference only:

• IETF Use Case Document

|Feature dependencies=* Integration of the newly released WebRTC code.

• Work product from IETF and W3 working groups
• Design of getUserMedia from Media Task Force
• Integration of Rob O'Callahan's Media API work.

|Feature requirements=Must Have:

• Security design review
• Security implementation review

• The ability to make a call from a web page using a machine's built-in mic and camera.
• Should be able to work with most (80-90%) of NAT setups across all geographies. Note that certain configurations will require a relay.
• Support for royalty-free voice-friendly audio codecs. Opus strongly preferred.
• Support for royalty-free video codecs
• Audio/Video/Data must be secure (note that no security is absolute)
• A Web API that's been sent to the public working groups for feedback.
• A Web API and transport for data that accompanies the audio and video.
• Congestion control (Automatic bandwidth adjustment) required to be in and debugged prior to Firefox's WebRTC code's fire release
• Ability to add/remove video from an active call and prior to a call being placed or received

Nice to Have:

• Video resolution switching
• If hardware support exists for encoding & decoding VP8, support for video on Mobile.

|Feature non-goals=We are not trying to support anything other than the items mentioned in this document. This page is primarily focused on what's required to ship the first phase of the feature. |Feature functional spec=* Audio codecs planned: G.711 & Opus. (Although royalty/license-free, we have no plans to support iLBC, iSAC and G.722)

• Video codecs planned: VP8

API Docs:

• Robert's MediaStream Processing API Proposal

|Feature ux design=Latest spec can be viewed here: people.mozilla.com/~jboriss/specs/webrtc_spec.png |Feature implementation plan=We created our first public demo at IETF 83 (Paris). We were able to demo video/audio calls and working data channels. We need to add signaling (JSEP) support and PeerConnection support. We’re creating a project planning map of work to be done and we have a WebRTC tracking bug, bug 665909 |Feature security review=It's important to realize that this is the first step in a plan to bring video to the platform. From both a security and privacy standpoint this means we'll need to be reviewing it with an eye towards an eye towards future use cases as well as what's scoped in this one page. |Feature privacy review=` |Feature localization review=` |Feature accessibility review=` |Feature qa review=` |Feature operations review=` |Feature implementation notes=` |Feature landing criteria=` }}

Feature details

Priority	P2
Rank	999
Theme / Goal	Connect
Roadmap	Gecko
Secondary roadmap	Platform
Feature list	Platform
Project	`
Engineering team	Media

{{#set:Feature priority=P2

|Feature rank=999 |Feature theme=Connect |Feature roadmap=Gecko |Feature secondary roadmap=Platform |Feature list=Platform |Feature project=` |Feature engineering team=Media }}

Team status notes

	status	notes
Products	`	This is not part of Kilimanjaro, but there is a strong desire to get the first phase released before the end of this year.
Engineering	`	`
Security	Reviews to start soon	`
Privacy	`	`
Localization	`	`
Accessibility	`	`
Quality assurance	`	`
User experience	Organizing meetings to discuss	`
Product marketing	`	`
Operations	`	`

{{#set:Feature products status=`

|Feature products notes=This is not part of Kilimanjaro, but there is a strong desire to get the first phase released before the end of this year. |Feature engineering status=` |Feature engineering notes=` |Feature security status=Reviews to start soon |Feature security health=` |Feature security notes=` |Feature privacy status=` |Feature privacy notes=` |Feature localization status=` |Feature localization notes=` |Feature accessibility status=` |Feature accessibility notes=` |Feature qa status=` |Feature qa notes=` |Feature ux status=Organizing meetings to discuss |Feature ux notes=` |Feature product marketing status=` |Feature product marketing notes=` |Feature operations status=` |Feature operations notes=` }}