Releases/Firefox 25/Test Plan
« Firefox 24 | Firefox 25 | Firefox 26 »
Firefox 25 Test Plan
Summary
The following is the test plan for Firefox 25 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 25. After the release this document will be used as an archive of what was done to validate this release.
Status | Currently in Aurora -- merging to Beta on September 19, 2013 |
Lead | Anthony Hughes (irc: ashughes) |
Builds | aurora.mozilla.org |
- If you have some free time, please pick a task below and get in touch with one of the leads.
Schedule
June 24, 2013 | Firefox 25 lands on Nightly |
August 5, 2013 | Firefox 25 moves up to Aurora |
September 19, 2013 | Firefox 25 moves up to Beta |
October 22, 2013 | Firefox 25 Release Candidate available for testing |
October 29, 2013 | Firefox 25 is released |
Aurora
Branch Migration
Pre Sign-off Tasks
- [DONE] Firefox branch migrated (Release Management)
- [DONE] Mozmill Tests branch migrated (Automation Development)
- [DONE] Updates throttled on aurora channel (Release Engineering)
- [DONE] Updates verified throttled on aurora channel (QA)
Sign-off Tasks
- [DONE] Mozmill Daily functional testruns pass with no unexpected failures/regressions (report)
- [DONE] Mozmill Daily l10n testruns pass with no unexpected failures/regressions (report)
- [DONE] Mozmill Daily endurance testruns pass with no unexpected failures/regressions (report)
- [DONE] Mozmill Daily remote testruns pass with no unexpected failures/regressions (report)
- [DONE] Mozmill Daily auroratest update testruns pass with no unexpected failures/regressions (report)
Post Sign-off Tasks
- [DONE] Aurora sign-off email sent (QA)
- [DONE] Updates unthrottled on aurora channel (Release Engineering)
- [DONE] Updates verified unthrottled on aurora channel (QA)
- [MISSED] Verify Stub Installers stage the new Aurora build (bug 910442)
Pre-Beta Sign-off
Checklist
- [DONE] Mozmill-CI automation testruns checked for potential blocking regressions
- [DONE] Features signed-off for Beta readiness, verified disabled if not
- [ON TRACK] WebRTC spotchecked for potential blocking regressions, parity with Firefox 22
- [ON TRACK] Flagged bug fixes verified
Automation
The following tracks the results of daily automation run against the Aurora builds as indicated by the Mozmill Addons, Endurance, Functional, L10n, Remote, and Update reports.
Top Failures | Platform | Instances | Bug | Status |
/testSecurity/testGreenLarry.js | All | 130 | bug 907101 | FIXED (2013-08-21) |
/testPreferences/testRestoreHomepageToDefault.js | All | 41 | bug 912363 | DISABLED (2013-09-05) |
/testPopups/testPopupsAllowed.js | All | 36 | bug 790218 | FIXED (2013-09-09) |
/testSecurity/testSafeBrowsingNotificationBar.js | All | 32 | bug 905033 | DISABLED (2013-08-12) |
/testFindInPage/testFindInPage.js | All | 24 | bug 909227 | BLOCKED by l10n bug 913396 (2013-09-05) |
/testSecurity/testDVCertificate.js | All | 24 | bug 890487 | FIXED (2013-08-06) |
/testSecurity/testMixedContentPage.js | All | 24 | bug 901951 | FIXED (2013-08-06) |
/testPrivateBrowsing/testAboutCache.js | Windows | 23 | bug 888924 | ASSIGNED (2013-09-05) |
/testPopups/testPopupsBlocked.js | Windows | 13 | n/a | WORKSFORME (2013-09-04) |
Resident Memory Usage | Week 1 | Week 2 | Week 3 | Week 4 | Week 5 | Week 6 | Net |
Windows XP | 105 MB | 106 MB | 106 MB | 106 MB | 106 MB | 104 MB | -0.952% |
Windows Vista | 97 MB | 97 MB | 97 MB | 97 MB | 97 MB | 97 MB | no change |
Windows 7 32-bit | 100 MB | 100 MB | 100 MB | 100 MB | 100 MB | 99 MB | -1.000% |
Windows 7 64-bit | 102 MB | 102 MB | 102 MB | 102 MB | 102 MB | 101 MB | -0.980% |
Windows 8 32-bit | 130 MB | 124 MB | 122 MB | 123 MB | 122 MB | 102 MB | -21.538% |
Windows 8 64-bit | 132 MB | 126 MB | 123 MB | 124 MB | 124 MB | 104 MB | -21.212% |
Mac OSX 10.6 | 138 MB | 134 MB | 134 MB | 134 MB | 134 MB | 132 MB | -4.348% |
Mac OSX 10.7 | 203 MB | 200 MB | 200 MB | 201 MB | 199 MB | 197 MB | -2.956% |
Mac OSX 10.8 | 195 MB | 189 MB | 190 MB | 190 MB | 191 MB | 192 MB | -1.538% |
Ubuntu LTS 32-bit | 103 MB | 100 MB | 100 MB | 100 MB | 98 MB | 98 MB | -4.854% |
Ubuntu LTS 64-bit | 135 MB | 131 MB | 131 MB | 131 MB | 131 MB | 129 MB | -4.444% |
Ubuntu Desktop 32-bit | 102 MB | 100 MB | 100 MB | 100 MB | 101 MB | 99 MB | -2.941% |
Ubuntu Desktop 64-bit | 135 MB | 131 MB | 131 MB | 131 MB | 132 MB | 130 MB | -3.704% |
Features
The following features are tracked and tested as indicated by the release notes, relnote+, Firefox, and Platform feature bugs.
- [DONE] Web Audio (Manuela Muntean)
- [DONE] Find Bar Redesign (Manuela Muntean)
- [DONE] Reset Firefox Saves Windows and Tabs (Ioana Budnar)
- [DONE] Reset Profile When Unused for Months (Simona Marcu)
- [DONE] CSS3 background-attachment:local Support (Petruta Rasa)
- [DONE] Larry Panel (Bogdan Maris)
- [DONE] Emacs Map#forEach and Set#forEach (Paul Silaghi)
- [DONE] ES6 math functions (Mihai Morar)
- [DONE] Support in-line HTML5 iframes (Alexandra Lucinet)
- [DONE] Session Restore backend work, tab close & restore (Cornel Ionce)
- [DROPPED] Background Thumbnail Service (Paul Silaghi)
- [DROPPED] Prototype DOM Gamepad API (Alexandra Lucinet)
WebRTC
Using the Moztrap test as a guideline, verify that Firefox 25.0a2 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:
- Have only one browser open on each machine at any given time
- Have only one call running at any given time on a particular machine
- Make sure the caller and callee are always on *different* machines.
- Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
- If you find regressions, report a bug and CC Randell Jesup and Maire Reavy; they can help track down if the regressions are real or not
- Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
Caller | Callee | Result |
Firefox 25.0a2 on Windows 7 | Firefox 25.0a2 on Windows 7 | PASS |
Firefox 22.0 on Windows 7 | Firefox 25.0a2 on Windows XP | FAIL (1*) |
Chrome Stable on Windows 7 | Firefox 25.0a2 on Windows 8 | PASS |
Firefox 25.0a2 on Windows 7 | Firefox 22.0 on Mac OSX 10.8 | PASS (2*) |
Firefox 25.0a2 on Windows 7 | Chrome Stable on Ubuntu 12.04 | PASS |
Firefox 22.0 on Windows XP | Firefox 25.0a2 on Windows 7 | FAIL (4*) |
Firefox 25.0a2 on Windows XP | Firefox 22.0 on Windows XP | FAIL (10*) |
Firefox 25.0a2 on Windows XP | Chrome Stable on Windows 8 | FAIL (5*) |
Chrome Stable on Windows XP | Firefox 25.0a2 on Mac OSX 10.8 | PASS |
Firefox 22.0 on Windows XP | Firefox 25.0a2 on Ubuntu 13.04 | FAIL (10*) |
Chrome Stable on Windows 8 | Firefox 25.0a2 on Windows 7 | FAIL (8*) |
Firefox 25.0a2 on Windows 8 | Chrome Stable on Windows XP | FAIL (7*) |
Firefox 22.0 on Windows 8 | Firefox 25.0a2 on Windows 8 | PASS |
Firefox 25.0a2 on Windows 8 | Firefox 22.0 on Windows 8 | PASS |
Firefox 22.0 on Windows 8 | Firefox 25.0a2 on Mac OSX 10.8 | FAIL (6*) |
Chrome Stable on Windows 8 | Firefox 25.0a2 on Ubuntu 13.04 | PASS |
Firefox 25.0a2 on Mac OSX 10.8 | Firefox 22.0 on Windows 7 | FAIL (9*) |
Chrome Stable on Mac OSX 10.8 | Firefox 25.0a2 on Windows XP | FAIL (9*) |
Firefox 22.0 on Mac OSX 10.8 | Firefox 25.0a2 on Windows 8 | FAIL (11*) |
Firefox 25.0a2 on Mac OSX 10.8 | Chrome Stable on Mac OSX 10.8 | PASS |
Firefox 25.0a2 on Mac OSX 10.8 | Firefox 22.0 on Ubuntu 13.04 | FAIL (9*) |
Firefox 25.0a2 on Ubuntu 12.04 | Chrome Stable on Windows 7 | PASS |
Firefox 22.0 on Ubuntu 12.04 | Firefox 25.0a2 on Windows XP | PASS |
Chrome Stable on Ubuntu 12.04 | Firefox 25.0a2 on Windows 8 | PASS |
Firefox 25.0a2 on Ubuntu 12.04 | Firefox 22.0 on Mac OSX 10.8 | PASS |
Firefox 25.0a2 on Ubuntu 12.04 | Chrome Stable on Ubuntu 13.04 | PASS |
Detailed Results
- (1*) The caller hears himself, as well. The callee hears delayed sound (1-2sec) after a few minutes
- not reproducible if caller uses latest Aurora 25.0a2.
- (2*) The caller got a crash after ~12 min, while opening a page in a new tab: https://crash-stats.mozilla.com/report/index/97cf87a4-284f-4cd0-9a17-ed6cf2130906
- one time occurrence, not related to WebRTC
- (4*) Both caller and callee hear delayed sound (1-2secs) - reproducible also if callee uses Firefox 23.0.1
- (5*) The caller hears delayed sound
- reproducible also with FF23.0.1 and 24b9; moreover, with 23.0.1 and 24b.9, the delay appears on both sides
- (6*) The callee hear himself, as well
- reproducible also with 23.0.1 and Firefox 24 beta 10
- (7*) Callee's conversation page crashes when caller attends
- reproducible with FF23.0.1 or 24b9 (used by caller)
- (8*) when caller connects, both attendances remain stuck in “Connecting…” mode
- working with 23.0.1 and 24b9
- (9*) when caller connects, both attendances remain stuck in “Connecting…” mode
- working with 23.0.1 and 24b9
- the problem is the TURN server which is giving a bogus address
- (10*) sometimes, the image and sound is choppy and there is a delayed sound: ~5 seconds at caller, ~10seconds at callee; callee hears himself, as well
- reproducible with 23.0.1
- (11*) attendants see themselves instead of seeing the other person in the call and there is no sound
- reproducible with 23.0.1
Bug Fix Verifications
The following bugs have been fixed and flagged for verification. At a minimum, any critical/blocker bug should be verified before migration to Beta.
5 Total; 0 Open (0%); 2 Resolved (40%); 3 Verified (60%);
Potential Blockers
The following bugs have been nominated to block this milestone from uplifting to Beta.
- no blockers currently
Beta 1
Build Info
- Builds: FTP, HTTP
- Build ID: 20130917123208
- Change Log
- Mozmill Configs
Check List
Criteria | Description | Status |
Automation | No unexpected test failures from Mozmill functional testruns | SIGNED-OFF (report) |
Regression Testing | No unexpected failures from manually smoketesting past regressions and new features | SIGNED-OFF (report) |
Bug Fix Verifications | No unexpected failures from manually verifying landed bug fixes | SIGNED-OFF (report) |
Stability Verification | No unexpected spikes in top crash report for this milestone | SIGNED-OFF (report) |
Updates on betatest channel | No unexpected failures from Mozmill update testruns on the betatest channel | SIGNED-OFF (report) |
Updates on releasetest channel | No unexpected failures from Mozmill update testruns on the releasetest channel | SIGNED-OFF (report) |
Updates on betatest channel | No unexpected failures from Mozmill update testruns on the beta channel | SIGNED-OFF (report) |
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Test Coverage
- Breakpad
- NVidia 3D-Vision
- WebRTC AppRTC 1:1 calling (subset of Aurora testing)
- DefaultKeyBinding.dict support on Mac OSX (Paul Silaghi)
- Web Audio (Manuela Muntean)
- Find Bar Redesign (Manuela Muntean)
- Reset Firefox Saves Windows and Tabs (Ioana Budnar)
- Reset Profile When Unused for Months (Simona Marcu)
- Larry Panel (Bogdan Maris)
- Session Restore backend work, tab close & restore (Cornel Ionce)
- Platform Coverage
- Windows XP
- Windows 7 64-bit
- Windows 8.1 Preview 64-bit
- Mac OSX 10.8
- Ubuntu 13.04 64-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- When you get to the WebRTC smoketest refer to the Special Coverage for WebRTC section below for coverage details
- If you encounter a bug, report it and add it to the Issues list below using the bug number template
- Special Coverage for WebRTC
- Caller Firefox/Platform -> Callee Firefox/Platform
- Firefox 25.0b1 on Windows XP -> Firefox 25.0b1 on Windows 7
- Firefox 25.0b1 on Windows 7 -> Firefox 25.0b1 on Windows 8
- Firefox 25.0b1 on Windows 8 -> Firefox 25.0b1 on Mac OSX 10.8
- Firefox 25.0b1 on Mac OSX 10.8 -> Firefox 25.0b1 on Ubuntu 13.04
- Firefox 25.0b1 on Ubuntu 13.04 -> Firefox 25.0b1 on Windows XP
Issues
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
Bug | Status |
bug 917818 - Findbar buttons remain in pressed state on Linux | NEW - does not block 25b1 |
bug 912975 - Windows and tabs are not always restored after Reseting Firefox | NEW - does not block 25b1 |
bug 860249 - [Mozmill] Exception "Sorry, cannot connect to jsbridge extension, port 24242" | NEW - does not block 25b1 |
bug 791195 - Session is not restored after Firefox crash | NEW - does not block 25b1 |
Beta 2
Build Info
- Builds: FTP, HTTP
- Build ID: 20130923194050
- Change Log
- Mozmill Configs
Check List
Criteria | Description | Status |
Automation | No unexpected test failures from Mozmill functional testruns | SIGNED-OFF (report) |
Regression Testing | No unexpected failures from manually smoketesting past regressions and new features | SIGNED-OFF (report) |
Bug Fix Verifications | No unexpected failures from manually verifying landed bug fixes | SIGNED-OFF (report) |
Stability Verification | No unexpected spikes in top crash report for this milestone | SIGNED-OFF (report) |
Updates on betatest channel | No unexpected failures from Mozmill update testruns on the betatest channel | SIGNED-OFF (report) |
Updates on releasetest channel | No unexpected failures from Mozmill update testruns on the releasetest channel | SIGNED-OFF (report) |
Updates on beta channel | No unexpected failures from Mozmill update testruns on the beta channel | SIGNED-OFF (report) |
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Test Coverage
- Breakpad
- MD5 Hash De-support
- Plug-in Compatibility
- Exploratory testing around Flash 11.9 Beta : etherpad
- Platform Coverage
- Windows Vista
- Windows 7 32-bit
- Windows 8 64-bit
- Mac OSX 10.7
- Ubuntu 12.04 64-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- If you encounter a bug, report it and add it to the Issues list below using the bug number template
Beta 3
Build Info
- Builds: FTP, HTTP
- Build ID: 20130926170421
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Web Audio, WebGL
- Platforms: Windows XP, Windows 7 64-bit, Windows 8.1 32-bit, Mac OSX 10.8, Ubuntu 13.04 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Beta 4
Build Info
- Builds: FTP, HTTP
- Build ID: 20131001024718
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Web Audio, WebGL
- Platforms: Windows XP, Windows Vista, Windows 8 64-bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Beta 6
Build Info
- Builds: FTP, HTTP
- Build ID: 20131007213254
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Web Audio, WebGL, TLS/SSL
- Platforms: Windows XP, Windows Vista, Windows 8 64-bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
- Exploratory testing: TLS/SSL etherpad
Beta 7
Build Info
- Builds: FTP, HTTP
- Build ID: 20131010180222
- Change Log
- Mozmill Configs
Check List
- [ON TRACK] Automation (report)
- [ON TRACK] Regression Testing (report)
- [ON TRACK] Updates on betatest channel (report)
- [ON TRACK] Updates on releasetest channel (report)
- [ON TRACK] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Bookmarks Library, Mixed Content Blocker, Add-on Compatibility
- Platforms: Windows 7 64-bit, Windows 8 32-bit, Windows 8.1 64-bit, Mac OSX 10.9, Ubuntu 12.04 32-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Bugs
Potential Blockers
Bug | Status |
bug 921457 [WebAudio] No sound is played when connecting a Delay Node | 2013-10-10: Pending fix verification |
bug 904620 [WebGL] Hangs in SurfaceStream_TripleBuffer_Async::WaitForCompositor | 2013-10-02: VERIFIED FIXED |
Issues Found in Testing
Add any new issues found or old issues reopened to the list below. These issues will be triaged and escalated by the release leads.
28 Total; 5 Open (17.86%); 17 Resolved (60.71%); 6 Verified (21.43%);
Top Crashes
Conduct daily and weekly stability checks and list any tracked bugs below. Any crash in the top-20 should have the topcrash keyword added and be nominated to track. Use bughunter to help with URL correlations.
- [ON TRACK]
- bug 925407 crash in npswf32_11_9_900_117.dll@0x345b8d -- Pending investigation, needs symbols from Adobe (2013-10-10)
- bug 920725 crash in nsHtml5StreamParser::WriteStreamBytes -- Pending uplift of bug 915905 (2013-10-10)
- bug 881636 crash in mozilla::a11y::DocAccessible::UpdateTree -- Fix pending uplift (2013-10-10)
- [DONE]
- bug 921323 crash in OpenGL@0x32af (MakeCurrent from OMTC) -- FIXED in Firefox 26 (2013-10-10)
- bug 898156 crash in mozilla::net::FTPChannelParent::OnStartRequest -- FIXED in Firefox 25 (2013-09-12)
- bug 800347 crash in mozilla::ipc::RPCChannel::OnMaybeDequeueOne -- FIXED in Firefox 25 (2013-09-26)
- [AT RISK]
- bug 812683 [Win7] startup crash in _d2d_clear_surface @ CContext::RestorePipelineStateImpl (d3d11.dll, KB2670838, Asus Gamer OSD) -- no actionable leads (2013-10-10)
- [DROPPED]
- bug 767343 crash in nsSupportsStringImpl::SetData with abort -- below topcrash threshold (2013-10-10)
QA Wanted
- [ON TRACK]
- bug 918612 [Mac] Firefox 24 doesn't start when home directory is on AFP network share -- Pending test environment from Steven Michaud (2013-10-10)
- [DONE]
- bug 913407 Rendering issue in Addon manager -- Assigned to developer (2013-10-10)
- bug 913043 [adbe 3627468] Flash 11.9 crash in F1987069282 -- FIXED in Flash 11.9.900.110
- bug 922875 Stencil test fails even if there is no stencil buffer in the current FBO -- Investigated and commented; the issue doesn't seem to be a regression
- [DROPPED]
- bug 919076 Saved files permissions set incorrectly on Linux and Mac -- untracked as low-priority edge-case (2013-09-25)
- bug 917011 Video on second screen gets displayed on first screen in fullscreen mode -- no actionable leads (2013-09-26)
- bug 886670 Crash in mozilla::gl::SharedSurface_GLTexture::Fence @ libGPUSupportMercury on citadel demo guided tour -- no actionable leads (2013-09-26)
- bug 698296 crash in js::SweepBackgroundThings @ js::gc::Arena::finalize -- no actionable leads (2013-10-07)
- All QAWANTED Bugs
Verifications
Pick a bug from this list and test to verify the issue has been fixed; be sure to test a known affected build to confirm you are able to reproduce the original issue as described. Reopen the bug if it is reproducible in versions with status flags set to fixed. Set the status flag to verified for the version you tested once you've verified the fix. Set the bug status to VERIFIED FIXED once the fix is verified against the target milestone.
- Verified Fixes
272 Total; 272 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Unconfirmed
Pick a bug from this list and try to reproduce the issue using the information provided. If you reproduce the issue, provide a description of the circumstances, set the status to NEW, and move the bug into a component which matches the issue (ie. a bookmarks issue would go in the Bookmarks component). If you can't reproduce the issue ask the reporter for more information by setting the need-info? flag to their email address.
- Confirmed Bugs
193 Total; 193 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Other Bugs
The following other queries should be triaged periodically, mostly by the QA Driver, and nominated for testing.
- regressionwindow-wanted: nominate for qawanted if testing is needed or remove regressionwindow-wanted if the request is no longer valid
- [ fixed]: nominate for verifyme if QA testing is needed, [easy verify] of community testing is needed, or [qa-] if no testing is needed