Releases/Firefox 25/Test Plan: Difference between revisions
< Releases
Jump to navigation
Jump to search
| Line 1,250: | Line 1,250: | ||
|- | |- | ||
| 1. https://www.facebook.com/ | | 1. https://www.facebook.com/ | ||
| | | PASS | ||
| | | | ||
| | | | ||
| Line 1,257: | Line 1,257: | ||
|- | |- | ||
| 2. http://www.youtube.com/ | | 2. http://www.youtube.com/ | ||
| | | PASS | ||
| | | | ||
| | | | ||
| Line 1,264: | Line 1,264: | ||
|- | |- | ||
| 3. https://login.yahoo.com/ | | 3. https://login.yahoo.com/ | ||
| | | PASS | ||
| | | | ||
| | | | ||
| Line 1,271: | Line 1,271: | ||
|- | |- | ||
| 4. https://www.linkedin.com/ | | 4. https://www.linkedin.com/ | ||
| | | PASS | ||
| | | | ||
| | | | ||
| Line 1,278: | Line 1,278: | ||
|- | |- | ||
| 5. https://login.live.com/ | | 5. https://login.live.com/ | ||
| | | PASS | ||
| | | | ||
| | | | ||
| Line 1,285: | Line 1,285: | ||
|- | |- | ||
| 6. https://twitter.com/ | | 6. https://twitter.com/ | ||
| | | PASS | ||
| | | | ||
| | | | ||
Revision as of 09:44, 13 November 2013
« Firefox 24 | Firefox 25 | Firefox 26 »
Firefox 25 Test Plan
Summary
The following is the test plan for Firefox 25 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 25. After the release this document will be used as an archive of what was done to validate this release.
| Status | Currently in Aurora -- merging to Beta on September 19, 2013 |
| Lead | Anthony Hughes (irc: ashughes) |
| Builds | aurora.mozilla.org |
- If you have some free time, please pick a task below and get in touch with one of the leads.
Schedule
| June 24, 2013 | Firefox 25 lands on Nightly |
| August 5, 2013 | Firefox 25 moves up to Aurora |
| September 19, 2013 | Firefox 25 moves up to Beta |
| October 22, 2013 | Firefox 25 Release Candidate available for testing |
| October 29, 2013 | Firefox 25 is released |
Aurora
Branch Migration
Pre Sign-off Tasks
- [DONE] Firefox branch migrated (Release Management)
- [DONE] Mozmill Tests branch migrated (Automation Development)
- [DONE] Updates throttled on aurora channel (Release Engineering)
- [DONE] Updates verified throttled on aurora channel (QA)
Sign-off Tasks
- [DONE] Mozmill Daily functional testruns pass with no unexpected failures/regressions (report)
- [DONE] Mozmill Daily l10n testruns pass with no unexpected failures/regressions (report)
- [DONE] Mozmill Daily endurance testruns pass with no unexpected failures/regressions (report)
- [DONE] Mozmill Daily remote testruns pass with no unexpected failures/regressions (report)
- [DONE] Mozmill Daily auroratest update testruns pass with no unexpected failures/regressions (report)
Post Sign-off Tasks
- [DONE] Aurora sign-off email sent (QA)
- [DONE] Updates unthrottled on aurora channel (Release Engineering)
- [DONE] Updates verified unthrottled on aurora channel (QA)
- [MISSED] Verify Stub Installers stage the new Aurora build (bug 910442)
Pre-Beta Sign-off
Checklist
- [DONE] Mozmill-CI automation testruns checked for potential blocking regressions
- [DONE] Features signed-off for Beta readiness, verified disabled if not
- [ON TRACK] WebRTC spotchecked for potential blocking regressions, parity with Firefox 22
- [ON TRACK] Flagged bug fixes verified
Automation
The following tracks the results of daily automation run against the Aurora builds as indicated by the Mozmill Addons, Endurance, Functional, L10n, Remote, and Update reports.
| Top Failures | Platform | Instances | Bug | Status |
| /testSecurity/testGreenLarry.js | All | 130 | bug 907101 | FIXED (2013-08-21) |
| /testPreferences/testRestoreHomepageToDefault.js | All | 41 | bug 912363 | DISABLED (2013-09-05) |
| /testPopups/testPopupsAllowed.js | All | 36 | bug 790218 | FIXED (2013-09-09) |
| /testSecurity/testSafeBrowsingNotificationBar.js | All | 32 | bug 905033 | DISABLED (2013-08-12) |
| /testFindInPage/testFindInPage.js | All | 24 | bug 909227 | BLOCKED by l10n bug 913396 (2013-09-05) |
| /testSecurity/testDVCertificate.js | All | 24 | bug 890487 | FIXED (2013-08-06) |
| /testSecurity/testMixedContentPage.js | All | 24 | bug 901951 | FIXED (2013-08-06) |
| /testPrivateBrowsing/testAboutCache.js | Windows | 23 | bug 888924 | ASSIGNED (2013-09-05) |
| /testPopups/testPopupsBlocked.js | Windows | 13 | n/a | WORKSFORME (2013-09-04) |
| Resident Memory Usage | Week 1 | Week 2 | Week 3 | Week 4 | Week 5 | Week 6 | Net |
| Windows XP | 105 MB | 106 MB | 106 MB | 106 MB | 106 MB | 104 MB | -0.952% |
| Windows Vista | 97 MB | 97 MB | 97 MB | 97 MB | 97 MB | 97 MB | no change |
| Windows 7 32-bit | 100 MB | 100 MB | 100 MB | 100 MB | 100 MB | 99 MB | -1.000% |
| Windows 7 64-bit | 102 MB | 102 MB | 102 MB | 102 MB | 102 MB | 101 MB | -0.980% |
| Windows 8 32-bit | 130 MB | 124 MB | 122 MB | 123 MB | 122 MB | 102 MB | -21.538% |
| Windows 8 64-bit | 132 MB | 126 MB | 123 MB | 124 MB | 124 MB | 104 MB | -21.212% |
| Mac OSX 10.6 | 138 MB | 134 MB | 134 MB | 134 MB | 134 MB | 132 MB | -4.348% |
| Mac OSX 10.7 | 203 MB | 200 MB | 200 MB | 201 MB | 199 MB | 197 MB | -2.956% |
| Mac OSX 10.8 | 195 MB | 189 MB | 190 MB | 190 MB | 191 MB | 192 MB | -1.538% |
| Ubuntu LTS 32-bit | 103 MB | 100 MB | 100 MB | 100 MB | 98 MB | 98 MB | -4.854% |
| Ubuntu LTS 64-bit | 135 MB | 131 MB | 131 MB | 131 MB | 131 MB | 129 MB | -4.444% |
| Ubuntu Desktop 32-bit | 102 MB | 100 MB | 100 MB | 100 MB | 101 MB | 99 MB | -2.941% |
| Ubuntu Desktop 64-bit | 135 MB | 131 MB | 131 MB | 131 MB | 132 MB | 130 MB | -3.704% |
Features
The following features are tracked and tested as indicated by the release notes, relnote+, Firefox, and Platform feature bugs.
- [DONE] Web Audio (Manuela Muntean)
- [DONE] Find Bar Redesign (Manuela Muntean)
- [DONE] Reset Firefox Saves Windows and Tabs (Ioana Budnar)
- [DONE] Reset Profile When Unused for Months (Simona Marcu)
- [DONE] CSS3 background-attachment:local Support (Petruta Rasa)
- [DONE] Larry Panel (Bogdan Maris)
- [DONE] Emacs Map#forEach and Set#forEach (Paul Silaghi)
- [DONE] ES6 math functions (Mihai Morar)
- [DONE] Support in-line HTML5 iframes (Alexandra Lucinet)
- [DONE] Session Restore backend work, tab close & restore (Cornel Ionce)
- [DROPPED] Background Thumbnail Service (Paul Silaghi)
- [DROPPED] Prototype DOM Gamepad API (Alexandra Lucinet)
WebRTC
Using the Moztrap test as a guideline, verify that Firefox 25.0a2 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:
- Have only one browser open on each machine at any given time
- Have only one call running at any given time on a particular machine
- Make sure the caller and callee are always on *different* machines.
- Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
- If you find regressions, report a bug and CC Randell Jesup and Maire Reavy; they can help track down if the regressions are real or not
- Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
| Caller | Callee | Result |
| Firefox 25.0a2 on Windows 7 | Firefox 25.0a2 on Windows 7 | PASS |
| Firefox 22.0 on Windows 7 | Firefox 25.0a2 on Windows XP | FAIL (1*) |
| Chrome Stable on Windows 7 | Firefox 25.0a2 on Windows 8 | PASS |
| Firefox 25.0a2 on Windows 7 | Firefox 22.0 on Mac OSX 10.8 | PASS (2*) |
| Firefox 25.0a2 on Windows 7 | Chrome Stable on Ubuntu 12.04 | PASS |
| Firefox 22.0 on Windows XP | Firefox 25.0a2 on Windows 7 | FAIL (4*) |
| Firefox 25.0a2 on Windows XP | Firefox 22.0 on Windows XP | FAIL (10*) |
| Firefox 25.0a2 on Windows XP | Chrome Stable on Windows 8 | FAIL (5*) |
| Chrome Stable on Windows XP | Firefox 25.0a2 on Mac OSX 10.8 | PASS |
| Firefox 22.0 on Windows XP | Firefox 25.0a2 on Ubuntu 13.04 | FAIL (10*) |
| Chrome Stable on Windows 8 | Firefox 25.0a2 on Windows 7 | FAIL (8*) |
| Firefox 25.0a2 on Windows 8 | Chrome Stable on Windows XP | FAIL (7*) |
| Firefox 22.0 on Windows 8 | Firefox 25.0a2 on Windows 8 | PASS |
| Firefox 25.0a2 on Windows 8 | Firefox 22.0 on Windows 8 | PASS |
| Firefox 22.0 on Windows 8 | Firefox 25.0a2 on Mac OSX 10.8 | FAIL (6*) |
| Chrome Stable on Windows 8 | Firefox 25.0a2 on Ubuntu 13.04 | PASS |
| Firefox 25.0a2 on Mac OSX 10.8 | Firefox 22.0 on Windows 7 | FAIL (9*) |
| Chrome Stable on Mac OSX 10.8 | Firefox 25.0a2 on Windows XP | FAIL (9*) |
| Firefox 22.0 on Mac OSX 10.8 | Firefox 25.0a2 on Windows 8 | FAIL (11*) |
| Firefox 25.0a2 on Mac OSX 10.8 | Chrome Stable on Mac OSX 10.8 | PASS |
| Firefox 25.0a2 on Mac OSX 10.8 | Firefox 22.0 on Ubuntu 13.04 | FAIL (9*) |
| Firefox 25.0a2 on Ubuntu 12.04 | Chrome Stable on Windows 7 | PASS |
| Firefox 22.0 on Ubuntu 12.04 | Firefox 25.0a2 on Windows XP | PASS |
| Chrome Stable on Ubuntu 12.04 | Firefox 25.0a2 on Windows 8 | PASS |
| Firefox 25.0a2 on Ubuntu 12.04 | Firefox 22.0 on Mac OSX 10.8 | PASS |
| Firefox 25.0a2 on Ubuntu 12.04 | Chrome Stable on Ubuntu 13.04 | PASS |
Detailed Results
- (1*) The caller hears himself, as well. The callee hears delayed sound (1-2sec) after a few minutes
- not reproducible if caller uses latest Aurora 25.0a2.
- (2*) The caller got a crash after ~12 min, while opening a page in a new tab: https://crash-stats.mozilla.com/report/index/97cf87a4-284f-4cd0-9a17-ed6cf2130906
- one time occurrence, not related to WebRTC
- (4*) Both caller and callee hear delayed sound (1-2secs) - reproducible also if callee uses Firefox 23.0.1
- (5*) The caller hears delayed sound
- reproducible also with FF23.0.1 and 24b9; moreover, with 23.0.1 and 24b.9, the delay appears on both sides
- (6*) The callee hear himself, as well
- reproducible also with 23.0.1 and Firefox 24 beta 10
- (7*) Callee's conversation page crashes when caller attends
- reproducible with FF23.0.1 or 24b9 (used by caller)
- (8*) when caller connects, both attendances remain stuck in “Connecting…” mode
- working with 23.0.1 and 24b9
- (9*) when caller connects, both attendances remain stuck in “Connecting…” mode
- working with 23.0.1 and 24b9
- the problem is the TURN server which is giving a bogus address
- (10*) sometimes, the image and sound is choppy and there is a delayed sound: ~5 seconds at caller, ~10seconds at callee; callee hears himself, as well
- reproducible with 23.0.1
- (11*) attendants see themselves instead of seeing the other person in the call and there is no sound
- reproducible with 23.0.1
Bug Fix Verifications
The following bugs have been fixed and flagged for verification. At a minimum, any critical/blocker bug should be verified before migration to Beta.
5 Total; 0 Open (0%); 2 Resolved (40%); 3 Verified (60%);
Potential Blockers
The following bugs have been nominated to block this milestone from uplifting to Beta.
- no blockers currently
Beta 1
Build Info
- Builds: FTP, HTTP
- Build ID: 20130917123208
- Change Log
- Mozmill Configs
Check List
| Criteria | Description | Status |
| Automation | No unexpected test failures from Mozmill functional testruns | SIGNED-OFF (report) |
| Regression Testing | No unexpected failures from manually smoketesting past regressions and new features | SIGNED-OFF (report) |
| Bug Fix Verifications | No unexpected failures from manually verifying landed bug fixes | SIGNED-OFF (report) |
| Stability Verification | No unexpected spikes in top crash report for this milestone | SIGNED-OFF (report) |
| Updates on betatest channel | No unexpected failures from Mozmill update testruns on the betatest channel | SIGNED-OFF (report) |
| Updates on releasetest channel | No unexpected failures from Mozmill update testruns on the releasetest channel | SIGNED-OFF (report) |
| Updates on betatest channel | No unexpected failures from Mozmill update testruns on the beta channel | SIGNED-OFF (report) |
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Test Coverage
- Breakpad
- NVidia 3D-Vision
- WebRTC AppRTC 1:1 calling (subset of Aurora testing)
- DefaultKeyBinding.dict support on Mac OSX (Paul Silaghi)
- Web Audio (Manuela Muntean)
- Find Bar Redesign (Manuela Muntean)
- Reset Firefox Saves Windows and Tabs (Ioana Budnar)
- Reset Profile When Unused for Months (Simona Marcu)
- Larry Panel (Bogdan Maris)
- Session Restore backend work, tab close & restore (Cornel Ionce)
- Platform Coverage
- Windows XP
- Windows 7 64-bit
- Windows 8.1 Preview 64-bit
- Mac OSX 10.8
- Ubuntu 13.04 64-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- When you get to the WebRTC smoketest refer to the Special Coverage for WebRTC section below for coverage details
- If you encounter a bug, report it and add it to the Issues list below using the bug number template
- Special Coverage for WebRTC
- Caller Firefox/Platform -> Callee Firefox/Platform
- Firefox 25.0b1 on Windows XP -> Firefox 25.0b1 on Windows 7
- Firefox 25.0b1 on Windows 7 -> Firefox 25.0b1 on Windows 8
- Firefox 25.0b1 on Windows 8 -> Firefox 25.0b1 on Mac OSX 10.8
- Firefox 25.0b1 on Mac OSX 10.8 -> Firefox 25.0b1 on Ubuntu 13.04
- Firefox 25.0b1 on Ubuntu 13.04 -> Firefox 25.0b1 on Windows XP
Issues
List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.
| Bug | Status |
| bug 917818 - Findbar buttons remain in pressed state on Linux | NEW - does not block 25b1 |
| bug 912975 - Windows and tabs are not always restored after Reseting Firefox | NEW - does not block 25b1 |
| bug 860249 - [Mozmill] Exception "Sorry, cannot connect to jsbridge extension, port 24242" | NEW - does not block 25b1 |
| bug 791195 - Session is not restored after Firefox crash | NEW - does not block 25b1 |
Beta 2
Build Info
- Builds: FTP, HTTP
- Build ID: 20130923194050
- Change Log
- Mozmill Configs
Check List
| Criteria | Description | Status |
| Automation | No unexpected test failures from Mozmill functional testruns | SIGNED-OFF (report) |
| Regression Testing | No unexpected failures from manually smoketesting past regressions and new features | SIGNED-OFF (report) |
| Bug Fix Verifications | No unexpected failures from manually verifying landed bug fixes | SIGNED-OFF (report) |
| Stability Verification | No unexpected spikes in top crash report for this milestone | SIGNED-OFF (report) |
| Updates on betatest channel | No unexpected failures from Mozmill update testruns on the betatest channel | SIGNED-OFF (report) |
| Updates on releasetest channel | No unexpected failures from Mozmill update testruns on the releasetest channel | SIGNED-OFF (report) |
| Updates on beta channel | No unexpected failures from Mozmill update testruns on the beta channel | SIGNED-OFF (report) |
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Test Coverage
- Breakpad
- MD5 Hash De-support
- Plug-in Compatibility
- Exploratory testing around Flash 11.9 Beta : etherpad
- Platform Coverage
- Windows Vista
- Windows 7 32-bit
- Windows 8 64-bit
- Mac OSX 10.7
- Ubuntu 12.04 64-bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- If you encounter a bug, report it and add it to the Issues list below using the bug number template
Beta 3
Build Info
- Builds: FTP, HTTP
- Build ID: 20130926170421
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Web Audio, WebGL
- Platforms: Windows XP, Windows 7 64-bit, Windows 8.1 32-bit, Mac OSX 10.8, Ubuntu 13.04 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Beta 4
Build Info
- Builds: FTP, HTTP
- Build ID: 20131001024718
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Web Audio, WebGL
- Platforms: Windows XP, Windows Vista, Windows 8 64-bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Beta 6
Build Info
- Builds: FTP, HTTP
- Build ID: 20131007213254
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Web Audio, WebGL, TLS/SSL
- Platforms: Windows XP, Windows Vista, Windows 8 64-bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
- Exploratory testing: TLS/SSL etherpad
Beta 7
Build Info
- Builds: FTP, HTTP
- Build ID: 20131010180222
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Bookmarks Library, Mixed Content Blocker, Add-on Compatibility
- Platforms: Windows 7 64-bit, Windows 8 32-bit, Windows 8.1 64-bit, Mac OSX 10.9, Ubuntu 12.04 32-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Beta 8
Build Info
- Builds: FTP, HTTP
- Build ID: 20131015052812
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Software update for root-owned files, Downloads (multiple files - small and large ones, cancel, retry, open folder, open downloaded executables), PDF Viewer (open multiple pdf files and interact with them), Session Restore after restarting from regular and safe modes, Windows 8.1 exploratory testing, Mac OSX 10.9 exploratory testing
- Platforms: Windows XP, Windows 8.1 64-bit, Mac OSX 10.9, Ubuntu 13.10 32-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Beta 9
Build Info
- Builds: FTP, HTTP
- Build ID: 20131017174213
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Plug-in compatibility, Session Restore with multiple windows and tabs (exploratory testing with different options).
- Platforms: Windows XP, Windows 7 32-bit, Mac OSX 10.6, Ubuntu 12.04 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Beta 10
Build Info
- Builds: FTP, HTTP
- Build ID: 20131021191948
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Focus: Breakpad, Web Compatibility, PDF Viewer (open multiple pdf files and interact with them), Exit Firefox (close Firefox all possible ways when open with differently loaded profiles - many tabs, plugin content etc)
- Platforms: Windows XP, Windows 8.1 64-bit, Mac OSX 10.7, Ubuntu 13.10 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
Beta 11
Build Info
- Builds: FTP, HTTP
- Build ID: 20131022211721
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (see below)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- OCSP Disabled
- Use this Moztrap test as a guide
- Verify OCSP stapling is disabled and look for rendering, layout, and functional issues
- Check as many SSL enabled websites as possible, listing any issues found in the section below
- Note: the list below is a bare minimum, please add to it as time allows
| Sites Tested | Windows XP | Windows 7 32-bit | Windows 8 64-bit | Mac OSX 10.8 | Ubuntu 13.04 32-bit |
| Amazon | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Baidu | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Bit.ly | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Bugzilla | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Digg | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| EBay | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| Flickr | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| Outlook | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| QQ.com | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| Slashdot | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| SoundCloud | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Tumblr | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| Vimeo | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Wikipedia | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Wordpress | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Yahoo | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Yandex | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Youtube | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
Issues Found:
- No issues found
Fix Verifications
- [DONE] bug 929068 Tumblr logo does not show correctly at first load. (style sheet would not load at first load)
- [DONE] bug 929524 Layout is broken on twitter.com at initial loading
Beta 12
Build Info
- Builds: FTP, HTTP
- Build ID: 20131025150754
- Change Log
- Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Fix Verifications
- [DONE] bug 927901 Test has failed: Permission denied to access property 'encrypt'
- [DONE] bug 930827 Hang when visiting icu-project.org
Release Candidate
Build Info
- Builds:
build1build2build3 - Build ID:
201310211920592013102221540920131025151332 - Change Log:
build1build2build3 - Mozmill Configs
Check List
- [DONE] Automation (report)
- [DONE] Regression Testing (see below)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on release channel (report)
- [DONE] Verify update throttling level on release
- [DONE] Verify what's new page for 25.0-pl via bug 927975
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Past Regressions
- Focus: Breakpad, NVidia 3D-Vision, Partner repack update, and Windows share compatibility smoketests
- Platforms: Windows XP, Windows 7 64-bit, Windows 8.1 32-bit, Mac OSX 10.9, Ubuntu 13.04 64-bit
- Instructions: Run these tests and report bugs here; add any issues found to the Issues Found section below
- OCSP Disabled
- Use this Moztrap test as a guide
- Verify OCSP stapling is disabled and look for rendering, layout, and functional issues
- Check as many SSL enabled websites as possible, listing any issues found in the section below
- Note: the list below is a bare minimum, please add to it as time allows
| Sites Tested | Windows XP | Windows 7 64-bit | Windows 8.1 32-bit | Mac OSX 10.9 | Ubuntu 13.04 64-bit |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| SoundCloud | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Tumblr | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| [DONE] | [DONE] | [DONE] | [DONE] | [DONE] | |
| Yahoo | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Youtube | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| BT24 | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Google Mail | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| DuckDuckGo | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| MSN | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| PayPal | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Reuters | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| BBC News | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Moztrap | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Marketplace | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Mozilla Wiki | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Booking.com | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| London Stock Exchange | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Moneybookers | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Oracle Academy | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Sports Direct | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Emag | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Imdb | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
| Aol | [DONE] | [DONE] | [DONE] | [DONE] | [DONE] |
Issues Found:
- No issues found
Fix Verifications
- [DONE] bug 929068 Tumblr logo does not show correctly at first load. (style sheet would not load at first load)
- [DONE] bug 929524 Layout is broken on twitter.com at initial loading
- [DONE] bug 927901 Test has failed: Permission denied to access property 'encrypt'
- [DONE] bug 930827 Hang when visiting icu-project.org
- bug 927975 Enable Firefox /whatsnew for /pl (polish) locale on Firefox 25 -- once updates are released
25.0.1
Build Info
- Builds: build1
- Build ID: TBD
- Change Log: build1
- Mozmill Configs
Check List
- {{ |}} Automation (report)
- {{ |}} Regression Testing (see below)
- {{ |}} Updates on betatest channel (report)
- {{ |}} Updates on releasetest channel (report)
- {{ |}} Updates on release channel (report)
- {{ |}} Verify what's new page for 25.0-pl via bug 927975
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Past Regressions
- Focus: Breakpad
- Platforms: Windows XP, Windows 7 64-bit, Windows 8.1 32-bit, Mac OSX 10.9, Ubuntu 13.04 64-bit
- Instructions: Force a crash with the crashme extension and verify that the report is successfully submitted
- SSL Topsite Testing
- Navigate to a website which uses SSL and do some exploratory testing on that site
- Be sure to log in to the website if it has that functionality
- Report any issues with rendering and general usage, in particular if you get any SSL errors
- Suggestions for sites to test include email, banking, social networking, shopping, video streaming, and forum websites
- Try to test at least 25 different websites
- Add PASS if there were no problems or FAIL if you encountered an issue
- Be sure to note any issues in the Issues Found section below
| Sites Tested | Windows XP | Windows 7 64-bit | Windows 8.1 32-bit | Mac OSX 10.9 | Ubuntu 13.04 64-bit |
| 1. https://www.facebook.com/ | PASS | ||||
| 2. http://www.youtube.com/ | PASS | ||||
| 3. https://login.yahoo.com/ | PASS | ||||
| 4. https://www.linkedin.com/ | PASS | ||||
| 5. https://login.live.com/ | PASS | ||||
| 6. https://twitter.com/ | PASS | ||||
| 7. http://www.amazon.com/ | |||||
| 8. https://accounts.google.com/ | |||||
| 9. http://wordpress.com/ | |||||
| 10. http://www.ebay.com/ | |||||
| 11. https://www.tumblr.com/ | |||||
| 12. https://www.pinterest.com/ | |||||
| 13. https://www.paypal.com/ | |||||
| 14. https://instagram.com/accounts/login/ | |||||
| 15. https://login.alibaba.com/ | |||||
| 16. https://www.adcash.com/en/index.php | |||||
| 17. https://vimeo.com/ | |||||
| 18. http://www.aliexpress.com/ | |||||
| 19. https://ib.btrl.ro/BT24/ | |||||
| 20. https://www.pcgarage.ro | |||||
| 21. https://www.emag.ro/ | |||||
| 22. https://en.wikipedia.org/wiki/Main_Page | |||||
| 23. https://soundcloud.com/ | |||||
| 24. http://www.booking.com/ | |||||
| 25. http://www.bbc.com | |||||
| add more if time permits |
Issues Found:
- No issues found
Fix Verifications
- bug 933733 Page doesn't load/render if the mouse is not moving
- bug 935959 Update Mozilla to NSS 3.15.3 (new alternative NSS branch) to pick up a few fixes
Bugs
Potential Blockers
[ON TRACK]
- no tracked bugs
[DONE]
- bug 921457 [WebAudio] No sound is played when connecting a Delay Node
- bug 904620 [WebGL] Hangs in SurfaceStream_TripleBuffer_Async::WaitForCompositor
- bug 929068 Tumblr logo does not show correctly at first load. (style sheet would not load at first load)
- bug 929617 Validity period for stapled OCSP responses is too short
- bug 929524 Layout is broken on twitter.com at initial loading
- bug 927975 Enable Firefox /whatsnew for /pl (polish) locale on #Firefox25
[DROPPED]
- bug 923319 [WebAudio] Performance is extremely poor
- bug 923106 [WebAudio] Sound is not changed in the Oscillator audio source node
- bug 923058 [WebAudio] Jerky sound on Live Input node
Issues Found in Testing
Add any new issues found or old issues reopened to the list below. These issues will be triaged and escalated by the release leads.
33 Total; 5 Open (15.15%); 22 Resolved (66.67%); 6 Verified (18.18%);
Top Crashes
Conduct daily and weekly stability checks and list any tracked bugs below. Any crash in the top-20 should have the topcrash keyword added and be nominated to track. Use bughunter to help with URL correlations.
[ON TRACK]
- no tracked bugs
[DONE]
- bug 929009 crash in `anonymous namespace::downmix_to_stereo<float>(float*, long, float*, int)
- bug 925459 crashes related to bitguard.dll
- bug 921323 crash in OpenGL@0x32af (MakeCurrent from OMTC)
- bug 920725 crash in nsHtml5StreamParser::WriteStreamBytes
- bug 898156 crash in mozilla::net::FTPChannelParent::OnStartRequest
- bug 881636 crash in mozilla::a11y::DocAccessible::UpdateTree
- bug 873733 crash in npjp2 @ mozilla::plugins::parent::_retainobject with Java SE7
- bug 800347 crash in mozilla::ipc::RPCChannel::OnMaybeDequeueOne
[AT RISK]
- no bugs at risk
[DROPPED]
- bug 928020 crash in mozalloc_abort(char const* const) | xul.dll@... | js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>)
- bug 927081 crash in js::gc::IsAboutToBeFinalized<js::Shape>
- bug 925407 crash in npswf32_11_9_900_117.dll@0x345b8d
- bug 812683 [Win7] startup crash in _d2d_clear_surface @ CContext::RestorePipelineStateImpl (d3d11.dll, KB2670838, Asus Gamer OSD)
- bug 767343 crash in nsSupportsStringImpl::SetData with abort
QA Wanted
[ON TRACK]
- no bugs (list)
[DONE]
- bug 929063 Twitter layout broken/missing items
- bug 918612 [Mac] Firefox 24 doesn't start when home directory is on AFP network share
- bug 913407 Rendering issue in Addon manager
- bug 913043 [adbe 3627468] Flash 11.9 crash in F1987069282
- bug 922875 Stencil test fails even if there is no stencil buffer in the current FBO
[DROPPED]
- bug 919076 Saved files permissions set incorrectly on Linux and Mac
- bug 917011 Video on second screen gets displayed on first screen in fullscreen mode
- bug 886670 Crash in mozilla::gl::SharedSurface_GLTexture::Fence @ libGPUSupportMercury on citadel demo guided tour
- bug 698296 crash in js::SweepBackgroundThings @ js::gc::Arena::finalize
Verifications
- How to Verify a Fix
- Pick a bug from this list
- Download a build that is known to reproduce the bug and test to make sure you can see the bug
- Download a build that is said to be fixed and test to make sure you cannot see the bug
- If the bug reproduces in the fixed build:
- set the STATUS field to REOPENED and provide a comment detailing your testing
- If the bug does not reproduce in the fixed build:
- set the status flag to verified for the version you tested
- set the STATUS field to VERIFIED FIXED if the version tested is the same as the TARGET MILESTONE
- any bugs marked verified will automatically appear in the Verified Fixes section below
- Verified Fixes
272 Total; 272 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Unconfirmed
Pick a bug from this list and try to reproduce the issue using the information provided. If you reproduce the issue, provide a description of the circumstances, set the status to NEW, and move the bug into a component which matches the issue (ie. a bookmarks issue would go in the Bookmarks component). If you can't reproduce the issue ask the reporter for more information by setting the need-info? flag to their email address.
- Confirmed Bugs
187 Total; 187 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Other Bugs
The following other queries should be triaged periodically, mostly by the QA Driver, and nominated for testing.
- regressionwindow-wanted: nominate for qawanted if testing is needed or remove regressionwindow-wanted if the request is no longer valid
- [ fixed]: nominate for verifyme if QA testing is needed, [easy verify] of community testing is needed, or [qa-] if no testing is needed
New Platform Readiness
Windows 8.1
| Status | COMPLETE (due 16 October 2013) |
| Owner | Anthony Hughes |
| Manual Testing | COMPLETE (see known bugs listed below) |
| Automated Testing | COMPLETE (see known bugs listed below) |
| Bugs |
[ON TRACK]
[DONE]
|
Mac OSX 10.9
| Status | COMPLETE (due TBD October 2013) |
| Owner | Marcia Knous |
| Manual Testing | COMPLETE (see known bugs listed below) |
| Automated Testing | COMPLETE (integrated into Mozmill-CI workflow) |
| Bugs |
[ON TRACK]
[DONE]
|
Post-mortem
- Agenda & Notes
- Follow-up:
- OCSP - not tested well enough due to lack of live sites to test against
- QA needs to be able to test features even without live use cases
- Should Mozilla try to find a "launch partner" for these types of features in the future?
- WebCrypto - undocumented uplift to Firefox 25
- QA needs to be "involved" in uplift approvals, attending the RelMan meetings is one way we can achieve this
- App Manager - flew under the radar completely
- QA needs to escalate these "failures" as soon as they happen
- Talk to dkl or glob about identifying "features" based on Bugzilla heuristics (patch size, number of files, component, etc)
- l10n (24esr) - RelMan's mistake
- QA is incapable of identifying these issues via manual testing and automation
- RelEng should harden their system to mitigate exposure to human error
- QA should try to involve l10n more in community events, reach out to localizers more
- Mozmill-CI Instability
- Cannot continue to force European staff to troubleshoot when they are on PTO, sick, or should be sleeping
- A-Team needs someone in a North American timezone to maintain and troubleshoot issues during Beta/Release pushes
- OCSP - not tested well enough due to lack of live sites to test against