Releases/Firefox 17/Test Plan: Difference between revisions
< Releases
Jump to navigation
Jump to search
| Line 409: | Line 409: | ||
|- | |- | ||
| Windows 7 | | Windows 7 | ||
| | | Manuela | ||
| ''' | | '''IN PROGRESS''' - 2012-10-11 | ||
|- | |- | ||
| Windows 8 | | Windows 8 | ||
| Line 426: | Line 426: | ||
| Ubuntu 12.04 | | Ubuntu 12.04 | ||
| Manuela | | Manuela | ||
| ''' | | '''PASS''' - 2012-10-11 | ||
|- | |- | ||
| colspan="3" | '''Special Cases''' | | colspan="3" | '''Special Cases''' | ||
Revision as of 08:30, 11 October 2012
« Firefox 16 | Firefox 17 | Firefox 18 »
Firefox 17 Test Plan
Summary
| Milestone | Firefox 17 |
| Status | Currently in Aurora -- merging to Beta on October 9, 2012 |
| Lead | Anthony Hughes (irc: ashughes) |
| 2nd | Juan Becerra (irc: juanb) |
| Builds | [ftp] [http] [web] |
The following is the test plan for Firefox 17 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 17. After the release this document will be used as an archive of what was done to validate this release.
If you have some free time, please pick a task below and get in touch with one of the leads.
Details
The following is a description of the tasks which need to be completed throughout this release.
- 2012-07-16: {{|}} Nightly Merge
- 2012-08-27: {{|}} Aurora Merge
- 2012-10-08: {{|}} Pre-Beta feature sign-offs
- 2012-10-11: {{|}} Beta 1
- 2012-10-18: {{|}} Beta 2
- 2012-10-25: {{|}} Beta 3
- 2012-11-01: {{|}} Beta 4
- 2012-11-08: {{|}} Beta 5
- 2012-11-15: {{|}} Beta 6
- 2012-11-19: {{|}} Release Candidate
- 2012-11-20: {{|}} Final
Nightly
Merge
- Functional Automation - result
- Update Spotchecks - result
Features
See the Feature Release Tracking wiki for a general overview
Features
See the Feature Release Tracking wiki for a general overview
| Feature | Owner | Status | Tests |
| Improve display of location bar results | Virgil Dicu | On track | |
| Firefox Social Integration | Anthony Hughes | On track | |
| HTML Tree Editor | Mihaela Velimiroviciu | ||
| Silent Update not now prompt | Ioana Budnar | Release target removed. |
Endurance Automation
The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.
- What to Do
- Check the Charts
- If you see an anomalous event (ie. spike or dip), get the report for the specific event
- If the event happens twice in a row, file a bug under Mozmill Tests
- Notify Anthony Hughes of the issue
- Record the event in this wiki below
- Current Status (Mihaela Velimiroviciu) - Last Updated
Bugzilla query error
Query options must be valid JSON.1
Functional Automation
The purpose of this section is to track any potential functional regressions via results reported in the dashboard.
- What to Do
- Check the testrun reports
- If you see any failing tests, check that there are not any bugs on file for it already
- If not, file a bug under Mozmill Tests
- Notify Anthony Hughes of the issue
- Record the event in this wiki below
- Current Status (Vlad Maniac / Alex Lakatos) - Last Updated
Bug Triage
The purpose of this section is to triage UNCONFIRMED incoming bugs.
- What to Do
- Check the UNCONFIRMED bugs
- Test the bugs in the latest Firefox Nightly
- If you can't reproduce the bug, ask for the reporter to provide more information
- If you reproduce the bug, add your testing information and mark the bug NEW
- Notify Anthony Hughes of the NEW issue
- Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated
Aurora Merge
- NOTE - OSX 10.5 should not be able to install nor update to Firefox 17
| Builds | nightly |
| Changeset | tip (pushlog) |
- Checklist
- [DONE] Automated Testing
- [DONE] Manual Testing
- [DONE] Update Testing
- [DONE] Bug Triage
- [DONE] Peer Sign-off
- [ON TRACK] Final sign-off sent and updates qualified on Beta channel
Automated Testing
The following are the detailed results of the automated Mozmill test runs.
| Platform | PASS: report |
| Windows XP | PASS - 2012-08-28 |
| Windows Vista | PASS - 2012-08-28 |
| Windows 7 32-bit | PASS - 2012-08-28 |
| Windows 7 64-bit | PASS - 2012-08-28 |
| Windows 8 32-bit | N/A (pending IT) |
| Windows 8 64-bit | N/A (pending IT) |
| Mac OS X 10.6 | PASS - 2012-08-28 |
| Mac OS X 10.7 | PASS - 2012-08-28 |
| Mac OS X 10.8 | PASS - 2012-08-28 |
| Ubuntu 12.04 32-bit | PASS - 2012-08-28 |
| Ubuntu 12.04 64-bit | PASS - 2012-08-28 |
- Test Failures (known bugs)
Manual Testing
| Operating System | Tester | Plug-in Compatibility | Web Compatibility | OSX 10.5 EOL |
| Windows XP | Simona | PASS - 2012-08-29 | PASS - 2012-08-29 | N/A |
| Windows 7 | Ioana | PASS - 2012-08-29 | PASS - 2012-08-29 | N/A |
| Windows 8 | Paul | PASS - 2012-08-30 | PASS - 2012-08-30 | N/A |
| Mac OS X | VladG | PASS - 2012-08-30 | PASS - 2012-08-30 | PASS - 2012-08-30 |
| Ubuntu 12.04 | Ioana | PASS - 2012-08-30 | PASS - 2012-08-30 | N/A |
Plug-in Compatibility
Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.
- Adobe Flash Player
- 11.4 Release (make sure you can autoupdate)
- Microsoft Silverlight
- RealPlayer
- QuickTime
Web Compatibility
Test the following websites for rendering issues, functional issues, performance issues, and crashes
- Overall Top 10
- Top by Country (pick a random site from three random countries)
- Top by Category (pick a random site from three random categories)
- Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
- Try Google's MapsGL demo
- Try YouTube's HTML5 demo
OSX 10.5 EOL
Verify the following:
- Mac OSX 10.5 cannot download Firefox 17 (I'm not sure if this is scoped but try it anyway)
- Mac OSX 10.5 cannot start Firefox 17
- Mac OSX 10.5 cannot update to Firefox 17
- The build can be downloaded and installed
- Aurora didn't start, showing a incompatibility message
- The update channel seems to be close so I didn't manage to perform an update
Feature Sign-offs
- Log in to moztrap.mozilla.org
- Select Run Tests
- Select Product: Desktop Firefox
- Select Versions: 17
- Select Runs: Firefox 17 Aurora Merge Feature sign off
| Feature | Owner | Win32 | Mac | Linux |
| Improve display of location bar results | Virgil Dicu | PASS | PASS | PASS |
| Firefox Social Integration | Anthony Hughes | PASS | PASS | PASS |
- Details
Report any detailed results or bugs here...
Update Testing
Spotcheck updates from a previous Aurora to Aurora 17.
| Platform | Auroratest | Aurora |
| Windows 7 64-bit | PASS - 2012-08-30 | PASS - 2012-08-30 |
| Windows 8 64-bit | PASS - 2012-08-30 | PASS - 2012-08-30 |
| Mac OS X 10.7 | PASS - 2012-08-30 | PASS - 2012-08-30 |
| Ubuntu 12.04 64-bit | PASS - 2012-08-30 | PASS - 2012-08-30 |
- Issues
Bug Triage
- VERIFYME (5): verify the fix and do some exploratory testing around the issue
- FIXED (139): triage these bugs and test to ensure the fix is effective
- QAWANTED (6): triage these bugs and test to fulfill the QA request
- UNCONFIRMED (8): triage these bugs for potential new regressions
- Results
- VERIFIED: 24/139
- WONTVERIFY: 3/139
- HELPWANTED: 9/139
Beta 1
| Builds | |
| ID | |
| Changeset | 6b83222781e3 (pushlog) |
- Advisory
RelEng issues in generating the builds (see bug 796959 comment 5)
- Checklist
- [ON TRACK] Automated Testing
- [ON TRACK] Manual Testing
- {{ |}} Update Testing
- [ON TRACK] Bug Triage
- {{ |}} Peer Sign-off
- {{ |}} Final sign-off sent and updates qualified on Beta channel
Automated Testing
The following are the detailed results of the automated Mozmill test runs.
| Platform | Build #1 Results (report) | Build #2 Results (report) |
| Windows XP | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Windows Vista | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Windows 7 32-bit | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Windows 7 64-bit | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Windows 8 32-bit | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Windows 8 64-bit | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Mac OS X 10.6 | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Mac OS X 10.7 | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Mac OS X 10.8 | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Ubuntu 12.04 32-bit | PASS - 2012-10-09 | PASS - 2012-10-10 |
| Ubuntu 12.04 64-bit | PASS - 2012-10-09 | PASS - 2012-10-10 |
- Test Failures (known bugs)
Manual Testing
Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.
| Platform | Tester | Result |
| Windows 7 | Manuela | IN PROGRESS - 2012-10-11 |
| Windows 8 | Virgil | IN PROGRESS - 2012-10-10 |
| Mac OSX 10.7 | Ioana | In progress - 2012-10-10 |
| Mac OSX 10.8 | Virgil | result - YYYY-MM-DD |
| Ubuntu 12.04 | Manuela | PASS - 2012-10-11 |
| Special Cases | ||
| Mac OSX 10.5 de-support | tester | result - YYYY-MM-DD |
| What's New Page (requires updates on releasetest) | tester | result - YYYY-MM-DD |
Update Testing
- Partial:
- Complete:
| Platform | Releasetest ([ report]) | Beta ([ report]) |
| Windows XP | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windows Vista | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windows 7 32-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windows 7 64-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windows 8 32-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windows 8 64-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Mac OS X 10.6 | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Mac OS X 10.7 | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Mac OS X 10.8 | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Ubuntu 12.04 32-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Ubuntu 12.04 64-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
- What's New Page
Make sure the regression test passes
- Issues
Feature Sign-off
Feature owners to test and sign-off overall quality of their features following uplift to Beta.
| Feature | Owner | Win32 | Mac | Linux |
| Improve display of location bar results | Virgil Dicu | RESULTS - yyyy-mm-dd | RESULTS - yyyy-mm-dd | RESULTS - yyyy-mm-dd |
| HTML tree editor | Mihaela Velimiroviciu | RESULTS - yyyy-mm-dd | RESULTS - yyyy-mm-dd | RESULTS - yyyy-mm-dd |
| Firefox Social Integration | Anthony Hughes | n/a - delayed to Beta 2 | n/a - delayed to Beta 2 | n/a - delayed to Beta 2 |
Build #2 Verifications
- VERIFIED FIXED bug 799952
- VERIFIED FIXED bug 720619
- RESOLVED FIXED bug 798045 (could not verify because could not be reproduced)
Bugs
Triage
- VERIFYME: verify the fix and do some exploratory testing around the issue
- FIXED: triage these bugs and test to ensure the fix is effective
- QAWANTED: triage these bugs and test to fulfill the QA request
- UNCONFIRMED: triage these bugs for potential new regressions
New Issues
| ID | Summary | Priority | Status |
|---|---|---|---|
| 799856 | crash in mozilla::InternalInvalidateThebesLayersInSubtree , Firefox17b1 crashes when I canceled login to Facebook | -- | VERIFIED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Known Issues
12 Total; 1 Open (8.33%); 10 Resolved (83.33%); 1 Verified (8.33%);