QA/Desktop Firefox/WebRTC: Difference between revisions

Latest revision as of 19:27, 7 February 2014

See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing.

Basic Topology

The basic topology should include two separate machines on separate networks, each with their own NAT. We have some flexibility about which NAT, but for concreteness, I suggest a pair of Apple Airport Express NATs. I.e., the following topology:

The following test series should be run, with one machine on each network with Nightly.

Endpoints

Firefox Nightly <-> Firefox Nightly
Firefox Nightly <-> Chrome Stable
Firefox Nightly <-> Chrome Canary

Platforms

Desktop (Windows, Mac, Linux)
B2G (Unagi, Peak)
Android

AppRTC

Conduct a call for no less than 5 minutes with apprtc resulting in no lag, no freezing, no audio or video delays, no degradation in quality from the previous revision.

WebRTC.ME

Conduct a call for no less than 1 minute using webrtcme resulting in minimal lag and freezing.

WebRTC Landing

Run the WebRTC-Landing tests with no failures.

Known Issues

bug 936034 - Sound is distorted on one side on 1:1 call

Meetings

August 14, 2013: Kick-off meeting with WebRTC Devs

@@ Line 1: / Line 1: @@
 See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing.
-== Routine Manual Testing Tasks ==
+== Basic Topology ==
-The following tasks are essentially mechanical, but at least for now the
-need a person.
 The basic topology should include two separate machines on separate networks,
 each with their own NAT. We have some flexibility about which NAT, but
@@ Line 15: / Line 11: @@
 The following test series should be run, with one machine on each network with Nightly.
-Full test series on AppRTC
+=== Endpoints ===
+* Firefox Nightly <-> Firefox Nightly
-* A call on [http://apprtc.webrtc.org apprtc] between each pair of desktop endpoints
+* Firefox Nightly <-> Chrome Stable
-** (9 calls in all)
+* Firefox Nightly <-> Chrome Canary
-* A call on [http://apprtc.webrtc.org apprtc] between Firefox and Chrome (Stable + Canary) on any platform
-** (2 calls in all).
-* Each call should last no less than 5 minutes with no major failures (failure to connect, sustained freezes, etc.)
-Connect series on WebRTCMe
-* A call on  [http://webrtcme.herokuapp.com/r/ webrtcme]  between two Firefox endpoints (any platform)
-* A call on  [http://webrtcme.herokuapp.com/r/ webrtcme] between Firefox and Chrome Stable (and platform)
-* Each call should connect and work for at least one minute
-B2G Testing
+=== Platforms ===
-* A call on  [http://webrtcme.herokuapp.com/r/ webrtcme] between a B2G phone and any desktop OS for the following phones;
+* Desktop (Windows, Mac, Linux)
-** Unagi, Peak
+* B2G (Unagi, Peak)
-* This test needs to be run on a home-built build with Steven's patch
+* Android
-* Each call should connect and work for at least one minute with no major glitches (some video lag and freezing is acceptable).
-We need some Android tests, but I don't know what the status of Android is, so I'm reluctant to prescribe any.
+=== AppRTC ===
+Conduct a call for no less than 5 minutes with [http://apprtc.webrtc.org apprtc] resulting in no lag, no freezing, no audio or video delays, no degradation in quality from the previous revision.
-* Additional tests:
+=== WebRTC.ME ===
-  * WebRTC Landing tests  [http://mozilla.github.com/webrtc-landing WebRTC-Landing] on all platforms.
+Conduct a call for no less than 1 minute using [http://webrtcme.herokuapp.com/r/ webrtcme] resulting in minimal lag and freezing.
-== QA Tasks ==
+=== WebRTC Landing ===
+Run the [http://mozilla.github.com/webrtc-landing WebRTC-Landing tests] with no failures.
-* Daily verification of whatever fixes landed in that day's Nightly
+== Known Issues ==
-* Weekly triage and attempted confirmation of unconfirmed/regression bugs
+* {{bug|936034}} - Sound is distorted on one side on 1:1 call
-* Monthly re-evaluation of regression rate to inform adjustments to the above approach
-=== Testing ===
+== Meetings ==
-* Manual
+* [https://etherpad.mozilla.org/qa-webrtc-meeting-20130814 August 14, 2013]: Kick-off meeting with WebRTC Devs
-** What must be done manually?
-* Automated
-** What can be automated?