QA/Desktop Firefox/WebRTC: Difference between revisions

Latest revision as of 19:27, 7 February 2014

See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing.

Basic Topology

The basic topology should include two separate machines on separate networks, each with their own NAT. We have some flexibility about which NAT, but for concreteness, I suggest a pair of Apple Airport Express NATs. I.e., the following topology:

The following test series should be run, with one machine on each network with Nightly.

Endpoints

Firefox Nightly <-> Firefox Nightly
Firefox Nightly <-> Chrome Stable
Firefox Nightly <-> Chrome Canary

Platforms

Desktop (Windows, Mac, Linux)
B2G (Unagi, Peak)
Android

AppRTC

Conduct a call for no less than 5 minutes with apprtc resulting in no lag, no freezing, no audio or video delays, no degradation in quality from the previous revision.

WebRTC.ME

Conduct a call for no less than 1 minute using webrtcme resulting in minimal lag and freezing.

WebRTC Landing

Run the WebRTC-Landing tests with no failures.

Known Issues

bug 936034 - Sound is distorted on one side on 1:1 call

Meetings

August 14, 2013: Kick-off meeting with WebRTC Devs

@@ Line 1: / Line 1: @@
 See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing.
-== Routine Manual Testing Tasks ==
+== Basic Topology ==
-The following tasks are essentially mechanical, but at least for now the
-need a person.
 The basic topology should include two separate machines on separate networks,
 each with their own NAT. We have some flexibility about which NAT, but
@@ Line 13: / Line 9: @@
 [[File:topoplogy.png]]
-* Platforms to include all Windows, Mac OSX, Linux, Android, FirefoxOS
+The following test series should be run, with one machine on each network with Nightly.
-* Daily regression testing with Firefox Nightly across all platforms to include:
-** Calling with [http://apprtc.webrtc.org apprtc] using various combinations of Firefox and Chrome
+=== Endpoints ===
-** Calling with [http://webrtcme.herokuapp.com/r/ webrtcme] using various combinations of Firefox and Chrome
+* Firefox Nightly <-> Firefox Nightly
-** Evaluation of audio/video delay
+* Firefox Nightly <-> Chrome Stable
-** [http://mozilla.github.com/webrtc-landing WebRTC-Landing] tests pass
+* Firefox Nightly <-> Chrome Canary
-** ''Daily might be too ambitious, can it be done weekly?''
+=== Platforms ===
+* Desktop (Windows, Mac, Linux)
+* B2G (Unagi, Peak)
+* Android
+=== AppRTC ===
+Conduct a call for no less than 5 minutes with [http://apprtc.webrtc.org apprtc] resulting in no lag, no freezing, no audio or video delays, no degradation in quality from the previous revision.
+=== WebRTC.ME ===
+Conduct a call for no less than 1 minute using [http://webrtcme.herokuapp.com/r/ webrtcme] resulting in minimal lag and freezing.
-== QA Tasks ==
+=== WebRTC Landing ===
+Run the [http://mozilla.github.com/webrtc-landing WebRTC-Landing tests] with no failures.
-* Daily verification of whatever fixes landed in that day's Nightly
+== Known Issues ==
-* Weekly triage and attempted confirmation of unconfirmed/regression bugs
+* {{bug|936034}} - Sound is distorted on one side on 1:1 call
-* Monthly re-evaluation of regression rate to inform adjustments to the above approach
-=== Testing ===
+== Meetings ==
-* Manual
+* [https://etherpad.mozilla.org/qa-webrtc-meeting-20130814 August 14, 2013]: Kick-off meeting with WebRTC Devs
-** What must be done manually?
-* Automated
-** What can be automated?