Platform/Features/WebRTC: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
(Simplify the codec debate summary)
Line 20: Line 20:
}}
}}
{{FeaturePageBody
{{FeaturePageBody
|Feature open issues and risks=* VP8 is the only open, modern video codec under consideration.  There is heavy debate going on in the IETF working group about what video codec should be mandated by the spec.  Some are arguing that H.264 should be selected because the IP issues surrounding H.264 are unknown and handled by a consortium (MPE-LA) and VP8 is likely to face patent/licensing challenges that will bog down general adoption of the codec. Without a common video codec, the spec could fail to gain acceptance and adoption.
|Feature open issues and risks=* VP8 is the only open, modern video codec under consideration.  There is heavy debate going on in the IETF working group about what video codec should be mandated by the spec.  Some are arguing that H.264 should be selected because they prefer the patent licensing situation there (MPEG-LA) and more mature technology. Without a common video codec, the spec could fail to gain acceptance and adoption.
* Overall system latency (delay) might be an issue; if it is architectural changes may be needed. Testing will need to be done.  
* Overall system latency (delay) might be an issue; if it is architectural changes may be needed. Testing will need to be done.  
* Privacy UI/UX for this feature is a challenge since this involves camera and microphone data.  Security will be difficult due to the wish for applications to be able to provide useful interfaces, in conflict with the need to guarantee users are safe with regards to access to their camera and microphones. This design is being discussed with the security team.   
* Privacy UI/UX for this feature is a challenge since this involves camera and microphone data.  Security will be difficult due to the wish for applications to be able to provide useful interfaces, in conflict with the need to guarantee users are safe with regards to access to their camera and microphones. This design is being discussed with the security team.   
* Good congestion control is essential for real-time video/audio communication.  The algorithm for this is still being developed.  We can’t make our first release until we have a good algorithm implemented in the Firefox code.
* Good congestion control is essential for real-time video/audio communication.  The algorithm for this is still being developed.  We can’t make our first release until we have a good algorithm implemented in the Firefox code.
|Feature overview=This feature will allow people using the web platform to include video/audio conferencing and associated data as part of their websites and applications.  The main attributes include:
|Feature overview=This feature will allow people using the web platform to include video/audio conferencing and associated data as part of their websites and applications.  The main attributes include:
* Using the web as the setup end point for a "call."  This is different than other calling systems, which use SIP, XMPP or other signaling systems.
* Using the web as the setup end point for a "call."  This is different than other calling systems, which use SIP, XMPP or other signaling systems.
Line 30: Line 30:
* Video is displayed as a primary object in the browser and can be part of content.
* Video is displayed as a primary object in the browser and can be part of content.
* Data can be exchanged along with the video and audio.
* Data can be exchanged along with the video and audio.
|Feature users and use cases===Phase 1==
|Feature users and use cases===Phase 1==
We will support applications that do the following:
We will support applications that do the following:
Line 54: Line 53:


* [https://datatracker.ietf.org/doc/draft-ietf-rtcweb-use-cases-and-requirements/?include_text=1 IETF Use Case Document]
* [https://datatracker.ietf.org/doc/draft-ietf-rtcweb-use-cases-and-requirements/?include_text=1 IETF Use Case Document]
|Feature dependencies=* Integration of the newly released WebRTC code.
|Feature dependencies=* Integration of the newly released WebRTC code.
* Work product from IETF and W3 working groups
* Work product from IETF and W3 working groups
* Design of getUserMedia from Media Task Force  
* Design of getUserMedia from Media Task Force  
* Integration of Rob O'Callahan's Media API work.  
* Integration of Rob O'Callahan's Media API work.
|Feature requirements=Must Have:
|Feature requirements=Must Have:


Line 78: Line 76:
* Video resolution switching
* Video resolution switching
* If hardware support exists for encoding & decoding VP8, support for video on Mobile.
* If hardware support exists for encoding & decoding VP8, support for video on Mobile.
|Feature non-goals=We are not trying to support anything other than the items mentioned in this document.  This page is primarily focused on what's required to ship the first phase of the feature.
|Feature non-goals=We are not trying to support anything other than the items mentioned in this document.  This page is primarily focused on what's required to ship the first phase of the feature.
|Feature functional spec=* Audio codecs planned: G.711 & Opus.  (Although royalty/license-free, we have no plans to support iLBC, iSAC and G.722)
|Feature functional spec=* Audio codecs planned: G.711 & Opus.  (Although royalty/license-free, we have no plans to support iLBC, iSAC and G.722)
Line 86: Line 83:


* [http://hg.mozilla.org/users/rocallahan_mozilla.com/specs/raw-file/tip/StreamProcessing/StreamProcessing.html Robert's MediaStream Processing API Proposal]
* [http://hg.mozilla.org/users/rocallahan_mozilla.com/specs/raw-file/tip/StreamProcessing/StreamProcessing.html Robert's MediaStream Processing API Proposal]
|Feature implementation plan=We created our first public demo at IETF 83 (Paris).  We were able to demo video/audio calls and working data channels. We need to add signaling (JSEP) support and PeerConnection support.  We’re creating a project planning map of work to be done and we have a WebRTC tracking bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=webrtc bug 665909]
|Feature implementation plan=We created our first public demo at IETF 83 (Paris).  We were able to demo video/audio calls and working data channels. We need to add signaling (JSEP) support and PeerConnection support.  We’re creating a project planning map of work to be done and we have a WebRTC tracking bug, [https://bugzilla.mozilla.org/show_bug.cgi?id=webrtc bug 665909]
|Feature security review=It's important to realize that this is the first step in a plan to bring video to the platform. From both a security and privacy standpoint this means we'll need to be reviewing it with an eye towards an eye towards future use cases as well as what's scoped in this one page.
|Feature security review=It's important to realize that this is the first step in a plan to bring video to the platform. From both a security and privacy standpoint this means we'll need to be reviewing it with an eye towards an eye towards future use cases as well as what's scoped in this one page.

Revision as of 22:48, 23 April 2012

Please use "Edit with form" above to edit this page.

Status

WebRTC (formerly Video (and Audio) Conferencing)
Stage Planning
Status In progress
Release target Phase 1 - TBD (first release planned this year, 2012)
Health OK
Status note We have moved forward with development, even though the spec is still being hashed out. Our initial release may be prefixed.

{{#set:Feature name=WebRTC (formerly Video (and Audio) Conferencing)

|Feature stage=Planning |Feature status=In progress |Feature version=Phase 1 - TBD (first release planned this year, 2012) |Feature health=OK |Feature status note=We have moved forward with development, even though the spec is still being hashed out. Our initial release may be prefixed. }}

Team

Product manager Maire Reavy
Directly Responsible Individual Maire Reavy
Lead engineer Randell Jesup
Security lead Curtis Koenig (Maire to verify)
Privacy lead Sid Stamm (Maire to verify)
Localization lead `
Accessibility lead TBD
QA lead TBD
UX lead TBD (Jennifer Boriss created early wireframes for the project)
Product marketing lead TBD
Operations lead `
Additional members Cisco Team (Enda Mannion, Suhas Nandakumar, Ethan Hunt), Eric Rescorla, Rob O'Callahan (MediaStreams piece), Ralph Giles, TIm Terriberry, Anant Naranyanan (part time, mostly spec work after IETF 83)

{{#set:Feature product manager=Maire Reavy

|Feature feature manager=Maire Reavy |Feature lead engineer=Randell Jesup |Feature security lead=Curtis Koenig (Maire to verify) |Feature privacy lead=Sid Stamm (Maire to verify) |Feature localization lead=` |Feature accessibility lead=TBD |Feature qa lead=TBD |Feature ux lead=TBD (Jennifer Boriss created early wireframes for the project) |Feature product marketing lead=TBD |Feature operations lead=` |Feature additional members=Cisco Team (Enda Mannion, Suhas Nandakumar, Ethan Hunt), Eric Rescorla, Rob O'Callahan (MediaStreams piece), Ralph Giles, TIm Terriberry, Anant Naranyanan (part time, mostly spec work after IETF 83) }}

Open issues/risks

  • VP8 is the only open, modern video codec under consideration. There is heavy debate going on in the IETF working group about what video codec should be mandated by the spec. Some are arguing that H.264 should be selected because they prefer the patent licensing situation there (MPEG-LA) and more mature technology. Without a common video codec, the spec could fail to gain acceptance and adoption.
  • Overall system latency (delay) might be an issue; if it is architectural changes may be needed. Testing will need to be done.
  • Privacy UI/UX for this feature is a challenge since this involves camera and microphone data. Security will be difficult due to the wish for applications to be able to provide useful interfaces, in conflict with the need to guarantee users are safe with regards to access to their camera and microphones. This design is being discussed with the security team.
  • Good congestion control is essential for real-time video/audio communication. The algorithm for this is still being developed. We can’t make our first release until we have a good algorithm implemented in the Firefox code.

Stage 1: Definition

1. Feature overview

This feature will allow people using the web platform to include video/audio conferencing and associated data as part of their websites and applications. The main attributes include:

  • Using the web as the setup end point for a "call." This is different than other calling systems, which use SIP, XMPP or other signaling systems.
  • Allowing sites access to video camera feed for other purposes than a call, such as graphically manipulating the video feed or allowing users to record video
  • Once capabilities are exchanged, a point-to-point connection is established between browsers when possible. This includes NAT traversal.
  • Video is displayed as a primary object in the browser and can be part of content.
  • Data can be exchanged along with the video and audio.

2. Users & use cases

Phase 1

We will support applications that do the following:

  • Call a friend using a social API like Facebook. From the chat/online window the user can click on a person and start a video call with them in the browser. The video/audio/data should be securely transmitted.
  • Video conferencing / sharing slides from inside an app. The user calls someone from inside an app and is able to share files, images, slides, etc through the P2P connection as he/she is talking via the data channel.
  • Games that integrate video, audio and data sharing. For example a chess game where you see the video and hear the audio of your opponent and play the game live using WebRTC’s data channels.

Note: Basic Interoperation between Firefox's WebRTC code and Chrome's WebRTC code will be supported as part of phase 1. This should include video, audio, and data channels interop if Chrome supports data channels in time.

The following is a nice-to-have use case that we may or may not support in phase 1. If we don't support it in phase 1, we will support it in phase 2:

  • Remote assistance/co-pilot - you "look over the shoulder" of someone as they browse (optionally controlling the mouse/keyboard), while also talking with them over audio or audio+video.

Phase 2

  • Support for Android Mobile devices.
  • Probably support for B2G Mobile devices (TBD)


For reference only:

3. Dependencies

  • Integration of the newly released WebRTC code.
  • Work product from IETF and W3 working groups
  • Design of getUserMedia from Media Task Force
  • Integration of Rob O'Callahan's Media API work.

4. Requirements

Must Have:

  • Security design review
  • Security implementation review
  • The ability to make a call from a web page using a machine's built-in mic and camera.
  • Should be able to work with most (80-90%) of NAT setups across all geographies. Note that certain configurations will require a relay.
  • Support for royalty-free voice-friendly audio codecs. Opus strongly preferred.
  • Support for royalty-free video codecs
  • Audio/Video/Data must be secure (note that no security is absolute)
  • A Web API that's been sent to the public working groups for feedback.
  • A Web API and transport for data that accompanies the audio and video.
  • Congestion control (Automatic bandwidth adjustment) required to be in and debugged prior to Firefox's WebRTC code's fire release
  • Ability to add/remove video from an active call and prior to a call being placed or received

Nice to Have:

  • Video resolution switching
  • If hardware support exists for encoding & decoding VP8, support for video on Mobile.

Non-goals

We are not trying to support anything other than the items mentioned in this document. This page is primarily focused on what's required to ship the first phase of the feature.

Stage 2: Design

5. Functional specification

  • Audio codecs planned: G.711 & Opus. (Although royalty/license-free, we have no plans to support iLBC, iSAC and G.722)
  • Video codecs planned: VP8

API Docs:

6. User experience design

`

Stage 3: Planning

7. Implementation plan

We created our first public demo at IETF 83 (Paris). We were able to demo video/audio calls and working data channels. We need to add signaling (JSEP) support and PeerConnection support. We’re creating a project planning map of work to be done and we have a WebRTC tracking bug, bug 665909

8. Reviews

Security review

It's important to realize that this is the first step in a plan to bring video to the platform. From both a security and privacy standpoint this means we'll need to be reviewing it with an eye towards an eye towards future use cases as well as what's scoped in this one page.

Privacy review

`

Localization review

`

Accessibility

`

Quality Assurance review

`

Operations review

`

Stage 4: Development

9. Implementation

`

Stage 5: Release

10. Landing criteria

` {{#set:Feature open issues and risks=* VP8 is the only open, modern video codec under consideration. There is heavy debate going on in the IETF working group about what video codec should be mandated by the spec. Some are arguing that H.264 should be selected because they prefer the patent licensing situation there (MPEG-LA) and more mature technology. Without a common video codec, the spec could fail to gain acceptance and adoption.

  • Overall system latency (delay) might be an issue; if it is architectural changes may be needed. Testing will need to be done.
  • Privacy UI/UX for this feature is a challenge since this involves camera and microphone data. Security will be difficult due to the wish for applications to be able to provide useful interfaces, in conflict with the need to guarantee users are safe with regards to access to their camera and microphones. This design is being discussed with the security team.
  • Good congestion control is essential for real-time video/audio communication. The algorithm for this is still being developed. We can’t make our first release until we have a good algorithm implemented in the Firefox code.

|Feature overview=This feature will allow people using the web platform to include video/audio conferencing and associated data as part of their websites and applications. The main attributes include:

  • Using the web as the setup end point for a "call." This is different than other calling systems, which use SIP, XMPP or other signaling systems.
  • Allowing sites access to video camera feed for other purposes than a call, such as graphically manipulating the video feed or allowing users to record video
  • Once capabilities are exchanged, a point-to-point connection is established between browsers when possible. This includes NAT traversal.
  • Video is displayed as a primary object in the browser and can be part of content.
  • Data can be exchanged along with the video and audio.

|Feature users and use cases===Phase 1== We will support applications that do the following:

  • Call a friend using a social API like Facebook. From the chat/online window the user can click on a person and start a video call with them in the browser. The video/audio/data should be securely transmitted.
  • Video conferencing / sharing slides from inside an app. The user calls someone from inside an app and is able to share files, images, slides, etc through the P2P connection as he/she is talking via the data channel.
  • Games that integrate video, audio and data sharing. For example a chess game where you see the video and hear the audio of your opponent and play the game live using WebRTC’s data channels.

Note: Basic Interoperation between Firefox's WebRTC code and Chrome's WebRTC code will be supported as part of phase 1. This should include video, audio, and data channels interop if Chrome supports data channels in time.

The following is a nice-to-have use case that we may or may not support in phase 1. If we don't support it in phase 1, we will support it in phase 2:

  • Remote assistance/co-pilot - you "look over the shoulder" of someone as they browse (optionally controlling the mouse/keyboard), while also talking with them over audio or audio+video.

Phase 2

  • Support for Android Mobile devices.
  • Probably support for B2G Mobile devices (TBD)


For reference only:

|Feature dependencies=* Integration of the newly released WebRTC code.

  • Work product from IETF and W3 working groups
  • Design of getUserMedia from Media Task Force
  • Integration of Rob O'Callahan's Media API work.

|Feature requirements=Must Have:

  • Security design review
  • Security implementation review
  • The ability to make a call from a web page using a machine's built-in mic and camera.
  • Should be able to work with most (80-90%) of NAT setups across all geographies. Note that certain configurations will require a relay.
  • Support for royalty-free voice-friendly audio codecs. Opus strongly preferred.
  • Support for royalty-free video codecs
  • Audio/Video/Data must be secure (note that no security is absolute)
  • A Web API that's been sent to the public working groups for feedback.
  • A Web API and transport for data that accompanies the audio and video.
  • Congestion control (Automatic bandwidth adjustment) required to be in and debugged prior to Firefox's WebRTC code's fire release
  • Ability to add/remove video from an active call and prior to a call being placed or received

Nice to Have:

  • Video resolution switching
  • If hardware support exists for encoding & decoding VP8, support for video on Mobile.

|Feature non-goals=We are not trying to support anything other than the items mentioned in this document. This page is primarily focused on what's required to ship the first phase of the feature. |Feature functional spec=* Audio codecs planned: G.711 & Opus. (Although royalty/license-free, we have no plans to support iLBC, iSAC and G.722)

  • Video codecs planned: VP8

API Docs:

|Feature ux design=` |Feature implementation plan=We created our first public demo at IETF 83 (Paris). We were able to demo video/audio calls and working data channels. We need to add signaling (JSEP) support and PeerConnection support. We’re creating a project planning map of work to be done and we have a WebRTC tracking bug, bug 665909 |Feature security review=It's important to realize that this is the first step in a plan to bring video to the platform. From both a security and privacy standpoint this means we'll need to be reviewing it with an eye towards an eye towards future use cases as well as what's scoped in this one page. |Feature privacy review=` |Feature localization review=` |Feature accessibility review=` |Feature qa review=` |Feature operations review=` |Feature implementation notes=` |Feature landing criteria=` }}

Feature details

Priority P2
Rank 999
Theme / Goal Connect
Roadmap Gecko
Secondary roadmap Platform
Feature list Platform
Project `
Engineering team Media

{{#set:Feature priority=P2

|Feature rank=999 |Feature theme=Connect |Feature roadmap=Gecko |Feature secondary roadmap=Platform |Feature list=Platform |Feature project=` |Feature engineering team=Media }}

Team status notes

  status notes
Products ` This is not part of Kilimanjaro, but there is a strong desire to get the first phase released before the end of this year.
Engineering ` `
Security Reviews to start soon `
Privacy ` `
Localization ` `
Accessibility ` `
Quality assurance ` `
User experience Organizing meetings to discuss `
Product marketing ` `
Operations ` `

{{#set:Feature products status=`

|Feature products notes=This is not part of Kilimanjaro, but there is a strong desire to get the first phase released before the end of this year. |Feature engineering status=` |Feature engineering notes=` |Feature security status=Reviews to start soon |Feature security health=` |Feature security notes=` |Feature privacy status=` |Feature privacy notes=` |Feature localization status=` |Feature localization notes=` |Feature accessibility status=` |Feature accessibility notes=` |Feature qa status=` |Feature qa notes=` |Feature ux status=Organizing meetings to discuss |Feature ux notes=` |Feature product marketing status=` |Feature product marketing notes=` |Feature operations status=` |Feature operations notes=` }}


Retrieved from "https://wiki.allizom.org/index.php?title=Platform/Features/WebRTC&oldid=423283"