QA/Desktop Firefox/WebRTC

See: https://wiki.mozilla.org/Media/WebRTC/Testing for background on automated testing.

Routine Manual Testing Tasks

The following tasks are essentially mechanical, but at least for now they need a person. The goal should be to find ways to automate this work.

Basic Topology

The basic topology should include two separate machines on separate networks, each with their own NAT. We have some flexibility about which NAT, but for concreteness, I suggest a pair of Apple Airport Express NATs. I.e., the following topology:

The following test series should be run, with one machine on each network with Nightly.

Endpoints

• Firefox Nightly <-> Firefox Nightly
• Firefox Nightly <-> Chrome Stable
• Firefox Nightly <-> Chrome Canary

Platforms

• Desktop (Windows, Mac, Linux)
• B2G (Unagi, Peak)
• Android?

AppRTC

Conduct a call for no less than 5 minutes with apprtc resulting in no lag or freezing.

Weekly Test Matrix Template

	Windows <version>	Mac OSX <version>	Linux <version>	Android <version>	B2G <version>
Windows <version>	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox
Mac OSX <version>	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox
Linux <version>	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox
Android <version>	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox
B2G <version>	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox	Firefox -> Chrome Chrome -> Firefox Firefox -> Firefox

<version> indicates Operating System version, a different version each week

WebRTC.ME

Conduct a call for no less than 1 minute using webrtcme resulting in minimal lag and freezing.

WebRTC Landing

Run the WebRTC-Landing tests with no failures.

Other Tasks

FIXED Bugs

The following tracks verification of WebRTC bug fixes. As you verify a bug fix please list it below.

• 549 more bugs

UNCONFIRMED Bugs

The following tracks confirmation of WebRTC bugs. As you confirm a bug exists please list it below.

• 17 more bugs

WebRTC Health Report

The following tries to estimate the relative "health" of WebRTC based on bugs in bugzilla. Update this table each week with the number of bugs reported, resolved, and reopened.

Period	Bugs Reported	Bugs Resolved	Bugs Reopened
2013.07.28 - 2013.08.03	9	32	4
2013.07-21 - 2013-07-27	14	17	3
2013.07.14 - 2013.07.20	12	11	2
2013.07.07 - 2013.07.13	16	16	2
2013.06.30 - 2013.07.06	11	17	3

Testing

AppRTC

• Demo Site: apprtc.webrtc.org
• Test Methodology:
  • Running the demo 1:1 with no other processes running
  • Running the demo 1:1 with another vanilla Firefox process running
  • Running the demo 1:1 with another heavy Firefox process running (Flash, videos, games, gmail, etc)
  • Restart the caller browser and repeat the above
  • Restart the callee browser and repeat the above
• Expected Result:
  • A call lasting 5 minutes or longer should be clear and accurate (no delays, no artifacts, no crashes or hangs)
• Platforms:
  • P1: Windows XP, Windows 7, Windows 8, Mac OSX 10.8, Ubuntu 12.04, B2G
  • P2: Windows Vista, Mac OSX 10.7, Ubuntu 13.04, Android
• Builds:
  • P1: Firefox Nightly, Chrome Stable, B2G Browser
  • P2: Firefox Aurora/Beta/Release, Chrome Canary, Firefox Android (all branches)

Results

Caller	Callee	Result
Windows XP w/Firefox Nightly YYYY.MM.DD	Windows 7 w/Firefox Nightly YYYY.MM.DD	RESULT

Activity Log

• YYYY.MM.DD: Description of what was tested, any bugs filed, etc