Releases/Firefox 24/Test Plan

From MozillaWiki
Jump to navigation Jump to search

« Firefox 23 | Firefox 24 | Firefox 25 »

Firefox 24 Test Plan

Summary

Milestone Firefox 24
Status Currently in Beta - to Release on Week of September 17, 2013
Lead Tracy Walker (irc: tracy)
2nd Anthony Hughes (irc: ashughes)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 24 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 24. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

  • 2013.05.14: [DONE] Nightly Merge
  • 2013.06.24: [DONE] Aurora Merge
  • 2013.08.02: [DONE] Pre-Beta feature sign-offs
  • 2013.08.08: [DONE] Beta 1
  • 2013.08.13: [DONE] Beta 2
  • 2013.08.15: [DONE] Beta 3
  • 2013.08.20: [DONE] Beta 4
  • 2013.08.22: [DONE] Beta 5
  • 2013.08.27: [DONE] Beta 6
  • 2013.08.29: [DONE] Beta 7
  • 2013.09.03: [DONE] Beta 8
  • 2013.09.05: [DONE] Beta 9
  • 2013.09.10: [DONE] Beta 10
  • 2013.09.13: [ON TRACK] Release Candidate
  • 2013.09.17: [ON TRACK] Final

Features

See the Feature Release Tracking wiki for a general overview

Feature Owner Status Tests

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Endurance Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihai Morar) - Last Updated - 2013.07.26
Full Query
ID Summary Priority Status
887791 Higher memory consumption on Mac 10.7.5 and 10.8.4 since June the 22nd -- VERIFIED

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);


Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Tracy Walker of the issue
  5. Record the event in this wiki below
Current Status (SoftVision) - Last Updated - yyyy-mm-dd

Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Tracy Walker of the NEW issue
Current Status (SoftVision) - Last Updated

Result Spreadsheet

Bug Fix Verifications

Bugzilla query error

Query options must be valid JSON.1


Known Issues

Full Query
ID Summary Priority Status
313959 On page print in landscape mode, the page is printed as portrait with cutoff -- NEW
694748 When I open Print Preview of the View Page Source, The Error console receives an error. -- RESOLVED
743563 Missing thumbnail previews on new tab page for not completely loaded URLs -- RESOLVED
759334 Print to file creates a file with unknown format -- RESOLVED
835319 crash in CMIDTable::GetLocalMIDEntry [win7] -- RESOLVED
872413 facebook messenger blank name list -- RESOLVED
881634 crash in mozilla::gfx::DrawTargetSkia::DrawSurface -- RESOLVED
882819 Re-enabling a Social provider enables as inactive -- RESOLVED
890247 The Post Privacy Setting dropdown of the Share panel is not fully displayed -- RESOLVED
901527 Audio static/"burble"/breakup in mozilla to mozilla webrtc calls -- RESOLVED
906057 Delay on WebRTC calls -- RESOLVED
907213 Small hangs on webRTC calls initiated from Linux P4 RESOLVED
907214 webRTC call freezes after 20 to 30 minutes on https://apprtc.webrtc.org/ page -- RESOLVED

13 Total; 1 Open (7.69%); 12 Resolved (92.31%); 0 Verified (0%);


Release Candidate

Build Info

Builds FTP, HTTP
Build ID 20130909203154
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [ON TRACK]
Automated Tests [ON TRACK]
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [ON TRACK]
Update Tests (releasetest) [ON TRACK]
Update Tests (beta) [ON TRACK]
Stability & Support sign-off [ON TRACK]

Regression Tests

Exploratory testing on the below areas:

  • Breakpad End-to-End functionality (use this MozTrap test for guidelines).
Platform Status (results)
Windows 7 64-bit [ON TRACK]
Windows 8 32-bit [ON TRACK]
Mac OSX 10.6 [ON TRACK]
Ubuntu 12.04 32-bit [ON TRACK]

Automated Tests

Test Result Criteria
Automated Tests results (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Locales: all locales

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		results (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b9 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		results (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.0.4
  • Builds: 20.0b1-24.0b9 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		results (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b9 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 400 [1] => Bad Request ) ) 1


Beta

Beta 10

Build Info

Builds FTP, HTTP
Build ID 20130909203154
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing on the below areas:

  • Breakpad End-to-End functionality (use this MozTrap test for guidelines).
  • Web compatibility exploratory smoketests
Platform Status (results)
Windows 7 64-bit [DONE]
Windows 8 32-bit [DONE]
Mac OSX 10.6 [DONE]
Ubuntu 12.04 32-bit [DONE]

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Locales: all locales

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b9 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.0.4
  • Builds: 20.0b1-24.0b9 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b9 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 400 [1] => Bad Request ) ) 1


Beta 9

Build Info

Builds FTP, HTTP
Build ID 20130905180733
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing on the below areas:

  • WebRTC (one-on-one calls from each OS to each OS, multiple attendants calls, Firefox to Firefox, Firefox to other browers etc).
    • The multiple attendants calls have delays, burbles, and background noise - according to Bug 902428 these issues are improved in the latest Nightly 26.0a1 and are not fixed in Firefox 24 beta.
  • Breakpad End-to-End functionality (use this MozTrap test for guidelines).
Platform Status (results)
Windows 7 32-bit [DONE]
Windows 8 64-bit [DONE]
Mac OSX 10.7 32-bit mode [DONE]
Ubuntu 12.04 64-bit [DONE]

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Locales: all locales

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b8 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.0.4
  • Builds: 20.0b1-24.0b8 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b8 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
913043 [adbe 3627468] Flash 11.9 crash in F1987069282 -- RESOLVED
913449 Contextual menu not visible around an iframe containing Flash content P5 RESOLVED
913474 Scrollbar overlaps the chat window -- RESOLVED

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);


Beta 8

Build Info

Builds FTP, HTTP
Build ID 20130902131354
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing on the below areas:

Platform Status (results)
Windows 7 32-bit [DONE]
Windows 8 64-bit [DONE]
Mac OSX 10.7 [DONE]
Ubuntu 64-bit [DONE]

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Locales: all locales

Failures (known)

  • test Failure bug 912023 - ' A new window has been opened' in /testPrivateBrowsing/testOpenClosePBKeyboardShortcut.js

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b7 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.0.4
  • Builds: 20.0b1-24.0b7 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b7 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
911981 Chat panels are lost while detaching/attaching -- VERIFIED
912110 crash in nsXPCWrappedJS::Release() with DatamngrHlpFF24.dll (Bandoo) -- RESOLVED

2 Total; 0 Open (0%); 1 Resolved (50%); 1 Verified (50%);


Beta 7

Build Info

Builds FTP, HTTP
Build ID 20130829135643
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing around the following areas:

  • Audio/Video (H.264 and MP3 decoding).
  • Content Security Policy (guidelines in bug csp and wiki - most of this feature is already tested automatically, so only bug work and exploratory testing is needed, http://csptesting.herokuapp.com/).
  • DevTools: Browser Console (errors, warnings, info, logs of all types).


Platform Status (results)
Windows XP [DONE]
Windows 7 64-bit [DONE]
Mac OSX 10.8 [DONE]
Ubuntu 32-bit [DONE]

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Locales: all locales

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b6 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.0.4
  • Builds: 20.0b1-24.0b6 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b6 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
911164 Current window loses focus temporarily when hovering the pointer over the QuickTime volume button P4 RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);


Beta 6

Build Info

Builds FTP, HTTP
Build ID 20130826142034
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing around the following areas:

  • New Tab page (all options/functionality)
  • Unified search preferences (Search bar, about:home, contextual menu option, URL bar)


Platform Status (results)
Windows XP [DONE]
Windows 7 64-bit [DONE]
Mac OSX 10.8 [DONE]
Ubuntu 32-bit [DONE]

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Locales: all locales

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b5 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.0.4
  • Builds: 20.0b1-24.0b5 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/13.04
  • Builds: 20.0b1-24.0b5 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
909700 DuckDuckGo Plus gives strange error message after installation on a clean profile -- RESOLVED
909715 DuckDuckGo Plus remains the selected engine in the search bar after changing it with Google and restarting Firefox -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Beta 5

Build Info

Builds FTP, HTTP
Build ID 20130822154523
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE] only de, en-US, fr, it run by automation
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing around the following areas:

  • SocialAPI (chat - chat, open/close/detach/rearrange multiple chat windows, comment, like, share etc)
  • Session Restore (manual and automated restore, restore at startup, restore after crash etc)
  • Printing (use as many printing options as possible)


Platform Status (results)
Windows Vista [DONE]
Windows 7 32-bit [DONE]
Mac OSX 10.6 [DONE]
Ubuntu 32-bit [DONE]

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: all locales

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		result (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
908639 Scrollbar overlapping the "Choose an emoticon" button in a chat window -- RESOLVED
908641 Chat area misplaced when the chat is detached -- RESOLVED
908642 Chat window is duplicated, after it was previously detached -- RESOLVED
908643 Last message and writing area are not visible in the chat window after re-attaching it -- RESOLVED
908646 Messages are hidden when detaching and attaching the window chat -- RESOLVED
908648 "Turn On" button remains disabled after pressing Oops, undo this! -- RESOLVED
908650 Chatting is still possible after turning the chat off -- RESOLVED
908651 The sidebar overlaps the chat window -- VERIFIED
908654 Chat window settings button misplaced upwards -- RESOLVED

9 Total; 0 Open (0%); 8 Resolved (88.89%); 1 Verified (11.11%);


Beta 4

Build Info

Builds FTP, HTTP
Build ID 20130819170952
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [DONE] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing around the following areas:

  • Crash Reporter spot checks (use this MozTrap test as a guideline)
  • WebRTC
    • Using the Moztrap test as a guideline, verify that Firefox 24.0b4 is no worse than Firefox 23.0.1 when making AppRTC calls of >5 minutes in length.
Caller Platform Callee Platform Caller Browser Callee Browser Result Notes/Issues
Windows XP Windows 7 Firefox 23.0.1 Firefox 24.0b4 [DONE] Caller: ~2s delay immediately
Callee: no issues
Windows XP Windows 8 Firefox 23.0b4 Firefox 23.0.1 [DONE] Caller: ~2s delay immediately
Callee: ~2s delay immediately
Windows 7 Windows XP Firefox 23.0.1 Firefox 24.0b4 [DONE] Caller: poor sound quality (unclear noise), delay (2s) after 5 minutes, increasing as time goes by
Callee: ~2s delay immediately
Windows 7 Mac OSX Chrome Stable Firefox 24.0b4 [DONE] Caller: no delay, poor sound quality
Callee: poor sound quality (unclear noise), no delay
Windows 7 Linux Firefox 24.0b4 Firefox 23.0.1 [DONE] Caller: no issues
Callee: no issues
Windows 8 Windows XP Firefox 24.0b4 Firefox 23.0.1 [DONE] Caller: ~1.5s delay immediately
Windows 8 Mac OSX Firefox 23.0.1 Firefox 24.0b4 [DONE] Caller: ~2s delay immediately
Callee: ~1.5s delay immediately
Windows 8 Linux Chrome Stable Firefox 24.0b4 [DONE] Caller: no issues
Callee: no issues
Mac OSX Windows 7 Firefox 24.0b4 Firefox 24.0b4 [DONE] Caller: no issues
Callee: no issues
Mac OSX Linux Firefox 23.0.1 Firefox 24.0b4 [DONE] Caller: no issues
Callee: no issues
Linux Windows 7 Firefox 24.0b4 Firefox 23.0.1 [DONE] Caller: increasing delay (2+ seconds) after 10 minutes
Callee: no delay, poor sound quality (unclear noise)
Linux Windows 8 Chrome Stable Firefox 24.0b4 [DONE] Caller: no delay, small hangs, good audio/video quality
Callee: small hangs if browsing other websites during chat, no delay, good audio/video quality
Linux Mac OSX Firefox 23.0.1 Firefox 24.0b4 [DONE] Caller: small hangs, poor sound quality (unclear noise), no delay
Callee: small hangs, no delay
*Note: at min 30, the call interrupted, the video blocked and audio stopped.

When testing, here are the guidelines to follow:

  • Have only one browser open on each machine at any given time
  • Have only one call running at any given time on a particular machine
  • Make sure the caller and callee are always on *different* machines.
  • Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
  • If testers find regressions during their testing, Randell and Maire can help track down if the regressions are real or not


Platform Status (results)
Windows Vista [DONE]
Windows 7 32-bit [DONE]
Mac OSX 10.6 [DONE]
Ubuntu 32-bit [DONE]

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW

Failures (known)

  • bug 907101 Test failure "Certificate's country code is displayed - '(US)' should equal '(US,postalCode=94041,STREET=650 Castro St Ste 300)'" in testGreenLarry.js was seen in every testrun.

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
907213 Small hangs on webRTC calls initiated from Linux P4 RESOLVED
907214 webRTC call freezes after 20 to 30 minutes on https://apprtc.webrtc.org/ page -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Beta 3

Build Info

Builds FTP, HTTP
Build ID 20130815180504
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing around the following areas:

  • Crash Reporter spot checks (use this MozTrap test as a guideline)
Platform Status (results)
Windows 7 64-bit [DONE]
Windows 8 32-bit [DONE]
Mac OSX 10.7 [DONE]
Ubuntu 64-bit [DONE]
Notes & Issues Found

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-24.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
880140 crash in nsDeviceContextSpecX::EndPage @ decode_data when printing -- VERIFIED

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);


Beta 2

Build Info

Builds FTP, HTTP
Build ID 20130812173056
Changes pushlog
Configs mozmill

Checklist

Task Status
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [ON TRACK] on going in beta cycles}
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Regression Tests

Exploratory testing around the following areas:

  • Mac OSX scrollbars (simple pages, rich content pages, long pages etc)
  • WebRTC (one-on-one calls, multiple attendants calls, Firefox to Firefox, Firefox to other browers etc)
  • PDF Viewer (open simple and complex PDFs, print PDF, complete PDF form etc)
  • Network Monitor (different protocols, status codes, cookies etc)
Platform Status
Windows 7 64-bit [DONE] PASS
Windows 8 32-bit [DONE] PASS
Mac OSX 10.7 [DONE] PASS
Ubuntu 12.10 64-bit [DONE] PASS

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-23.0b10 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-23.0b10 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-23.0b10 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
904561 [10.7] Scrollbars shouldn't grow on hover on OSX 10.7 (only on 10.8 and up) -- VERIFIED
904575 crash [@ libobjc.A.dylib@0xb150][@ libobjc.A.dylib@0xb158] -- RESOLVED
904598 WebRTC not working on Ubuntu and Mac OS X machines -- VERIFIED

3 Total; 0 Open (0%); 1 Resolved (33.33%); 2 Verified (66.67%);


Beta 1

Because of l10n repack issues on Windows, Beta 1 will ship with all locales on Mac and Linux but only en-US on Windows. Release August 9th of 24.0beta1build#2

Build Info

Builds FTP, HTTP
Build ID 20130806170643
Changes pushlog
Configs mozmill

Checklist

Task Status
Feature Sign-off [DONE]
Regression Tests [DONE]
Automated Tests [DONE]
Bug Fix Verifications [ON TRACK] on going in beta cycles
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Update Tests (beta) [DONE]
Stability & Support sign-off [DONE]

Feature Sign Off

Feature QA Contact Status
link to bug or feature page contact(s) [DONE] status

Notes

  • Web Audio has also been disabled on Firefox 24, after the pre-beta sign off.

Regression Tests

Moztrap Testrun #1913

Platform Status (results)
Windows 7 32-bit [DONE] PASS
Windows 8 64-bit [DONE] PASS
Mac OSX 10.8 [DONE] PASS
Ubuntu Linux 32-bit [DONE] PASS

Automated Tests

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW

Failures (known)

Update Tests

Test Result Criteria
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-23.0b10 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-23.0b10 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 20.0b1-23.0b10 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

New Issues

Full Query
ID Summary Priority Status
902463 This Connection is Untrusted message on facebook apps using clean profile -- RESOLVED
902468 Firefox gets stuck in Print Preview while trying to print a certain pdf file P3 RESOLVED
902486 Firefox doesn't start on Mac OS X using a remote profile from Windows -- RESOLVED

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);


Pre-beta Feature testing and Sign-off

  • Platforms: Windows XP/7/8, Ubuntu 13.04, Mac 10.7/10.8
Feature Owner Win32 Mac Linux
Web Audio Manuela Muntean PASS PASS PASS

Notes:

  • The following features have been disabled on Firefox 24:
    • Prototype DOM Gamepad API (Alexandra Lucinet)
    • Block cookies from sites I haven't visited (Bogdan Maris)

Aurora Merge

Builds nightlies
Changeset tip (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform
Windows XP [DONE] - 2013-06-27
Windows Vista [DONE] - 2013-06-27
Windows 7 32-bit [DONE] - 2013-06-27
Windows 7 64-bit [DONE] - 2013-06-27
Windows 8 32-bit [DONE] - 2013-06-27
Windows 8 64-bit [DONE] - 2013-06-27
Mac OS X 10.6 [DONE] - 2013-06-27
Mac OS X 10.7 [DONE] - 2013-06-27
Mac OS X 10.8 [DONE] - 2013-06-27
Ubuntu 12.04 32-bit [DONE] - 2013-06-27
Ubuntu 12.04 64-bit [DONE] - 2013-06-27
Test Failures (known bugs)

Update Testing

Spotcheck updates from a previous Aurora to Aurora 24.

Platform auroratest channel aurora channel
Windows 7 64-bit [DONE]

en-US 23.a2 20130527004019 - 20130627004018
da 23.a2 20130620004019 - 20130627004018

Mac OS X 10.7 [DONE]

22.a2 en-US 20130420004013 - 20130627004018
21.a2 it 20130401042013 - 20130627004018

Ubuntu 12.04 32-bit [DONE]

en-US 21.a2 20130317042012 - 20130627004018
pl 23.a2 20130620004019 - 20130627004018

Issues

Important Features/Bugs

Bug Triage

  1. VERIFYME: verify the fix and do some exploratory testing around the issue
  2. FIXED: triage these bugs and test to ensure the fix is effective
  3. QAWANTED: triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED: triage these bugs for potential new regressions

QAWANTED

Bugs requiring QA's assistance in debugging (steps, regression range, exploratory testing, verification, etc)


Nightly

Merge

Functional Automation - result
Update Spotchecks - result
Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Firefox_24/Test_Plan&oldid=707223"