Releases/Firefox 4RC/Test Plan: Difference between revisions
< Releases
| (42 intermediate revisions by 5 users not shown) | |||
| Line 10: | Line 10: | ||
As in previous milestones, we are going to run our automated tests suites, smoketests, basic functional tests, and updates tests on all shipped locales, using Mozmill, and we'll complement those with manual checks on en-US builds. In addition to the regular basic cycle of testing we'll focus on the following activities, some of which we have already done on previous betas, and some of which are specific to the release candidate: | As in previous milestones, we are going to run our automated tests suites, smoketests, basic functional tests, and updates tests on all shipped locales, using Mozmill, and we'll complement those with manual checks on en-US builds. In addition to the regular basic cycle of testing we'll focus on the following activities, some of which we have already done on previous betas, and some of which are specific to the release candidate: | ||
* We are going to verify [https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d322a1c8537e bug fixes since beta 12] and test around them with exploratory testing- Waverley/Smartware contractors | * We are going to verify [https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d322a1c8537e bug fixes since beta 12] and test around them with exploratory testing- Waverley/Smartware contractors - '''Done''' | ||
* [https://spreadsheets.google.com/ccc?hl=en&key=t0zrO3mJEMiQNc5YdpbzCxw&hl=en#gid=0 Plugins testing across platforms]: Flash, Silverlight, Quicktime, Adobe Reader (latest versions), Java, RealPlayer, Shockwave, and Windows Media Player (From the [https://addons.mozilla.org/en-US/firefox/browse/type:7 list of most popular plugins] in https://addons.mozilla.org/) - Waverley | * [https://spreadsheets.google.com/ccc?hl=en&key=t0zrO3mJEMiQNc5YdpbzCxw&hl=en#gid=0 Plugins testing across platforms]: Flash, Silverlight, Quicktime, Adobe Reader (latest versions), Java, RealPlayer, Shockwave, and Windows Media Player (From the [https://addons.mozilla.org/en-US/firefox/browse/type:7 list of most popular plugins] in https://addons.mozilla.org/) - Waverley - '''Done''' | ||
* Web Compatibility Testing (Round 3): Popular social, news, games, and entertainment sites (See [https://spreadsheets.google.com/ccc?key=0AvdU4bzekK2qdHk2eWNoZlNCZEstNnY3R0pBWGhEaVE&hl=en#gid=0 testing matrix]) - uTest | ** We should spot check Netflix - see marcia for testing account - '''Done''' | ||
* Hardware compatibility testing including blocklisting: Performed by outside vendor (See [https://wiki.mozilla.org/QA/Firefox_4/Test_Plan/Hardware_Compatibility_FF4_Test_Plan on-going]) - iBeta | * Web Compatibility Testing (Round 3): Popular social, news, games, and entertainment sites (See [https://spreadsheets.google.com/ccc?key=0AvdU4bzekK2qdHk2eWNoZlNCZEstNnY3R0pBWGhEaVE&hl=en#gid=0 testing matrix]) - uTest - '''Done''' | ||
* [https://addons.mozilla.org/en-US/firefox/compatibility/report/4.0 Top 50 add-ons] testing: basic functionality | * Hardware compatibility testing including blocklisting: Performed by outside vendor (See [https://wiki.mozilla.org/QA/Firefox_4/Test_Plan/Hardware_Compatibility_FF4_Test_Plan on-going]) - iBeta '''Done''' | ||
* A round of testing Zynga games through Facebook: Mafia Wars, FarmVille, FrontierVille, using accounts provided by Zynga - marcia | * [https://addons.mozilla.org/en-US/firefox/compatibility/report/4.0 Top 50 add-ons] testing: basic functionality - '''Done''' | ||
* Accessibility - MarcoZ | * A round of testing Zynga games through Facebook: Mafia Wars, FarmVille, FrontierVille, using accounts provided by Zynga - marcia, CantinaVille - '''Done''' | ||
* A cycle of top sites page load testing: top 20k sites - Tomcat | * Accessibility - MarcoZ - '''Signed off''' | ||
* Special areas sign-off: Sync (desktop and mobile) - Tracy/Tchung, Test Pilot/Feedback - juanb | * A cycle of top sites page load testing: top 20k sites - Tomcat - '''Done''' | ||
* Crash-stats stability assessment (after 3 full week days) - marcia | * Special areas sign-off: Sync (desktop and mobile) - Tracy/Tchung, Test Pilot/Feedback - juanb '''Signed off'''' | ||
* Major update and downgrade scenario assessment - Waverley/Smartware/juanb | * Crash-stats stability assessment (after 3 full week days) - marcia '''Pending results''' | ||
* Major update and downgrade scenario assessment - Waverley/Smartware/juanb - '''Done''' | |||
* Fx4 in-product web pages sign-off - marcia, juanb | * Fx4 in-product web pages sign-off - marcia, juanb | ||
** First Run, What's New, Release Notes, Getting Started, Snippet Server (about:home) | ** First Run, What's New, Release Notes, Getting Started, Snippet Server (about:home) | ||
* Web of Wonder Demos - Spotcheck for crashes and CPU spikes/usage - marcia, juanb | * [https://demos.mozilla.org/en-US/ Web of Wonder Demos] - Spotcheck for crashes and CPU spikes/usage - marcia, juanb - '''Done''' | ||
* Web properties sign-off: AMO, SUMO, Socorro, Input - WebQA | * Web properties sign-off: AMO, SUMO, Socorro, Input - WebQA | ||
* An RC-specific testday or possible meetup - QA team | * An RC-specific testday or possible meetup - QA team - '''Done''' | ||
* Test coverage of [https://spreadsheets.google.com/ccc?key=0AmkRt0ylPb8zdDVKNGQxUHBldDBHd0xQWHNzb2ticVE&hl=en&authkey=CMG_h-8M quit dialog] scenarios - ashughes, ''need others to help'', zpao on hand to answer questions | * Test coverage of [https://spreadsheets.google.com/ccc?key=0AmkRt0ylPb8zdDVKNGQxUHBldDBHd0xQWHNzb2ticVE&hl=en&authkey=CMG_h-8M quit dialog] scenarios - ashughes, ''need others to help'', zpao on hand to answer questions | ||
| Line 34: | Line 35: | ||
== Schedule == | == Schedule == | ||
When it's ready. | |||
== Builds == | == Builds == | ||
Builds for all | * Builds for RC2 (build3) candidates can be found: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/4.0rc2-candidates/build3/ here] | ||
* <strike>Builds for RC2 (build2) candidates can be found: [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/4.0rc2-candidates/build2/ here].</strike> | |||
* <strike>Builds for all RC1 candidates can be found [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/4.0rc1-candidates/ here].</strike> | |||
== Test Results == | |||
=== <strike>RC2 Build 3</strike> Final (Current) === | |||
* [[Releases/Firefox 4RC2build3/Test Plan:Smoketests | Automated Smoketests]] - ashughes: '''PASS''' | |||
* [[Releases/Firefox 4RC2build3/Test Plan:Updates | Automated Updates]] - ashughes: '''PASS''' | |||
* [[Releases/Firefox 4RC2build3/Test Plan:Major Update Smoketests | Major Update Smoketests]] - ashughes: '''PASS''' | |||
* [[Releases/Firefox 4RC2build3/Test Plan:Focus | Specific plan for RC2]] - team: '''result''' | |||
** Limited scope: Bug verifications and testing around fixes. | |||
* [https://spreadsheets.google.com/ccc?key=0AmkRt0ylPb8zdFZqV3NxeXBMYV94RXo4bVNiSmJJb2c&hl=en&authkey=CL7nj6MH Advertised Major Update] - ashughes/waverley: '''delayed''' | |||
** {{bug|646054}} - Empty update dialog for 3.6 advertised updates only | |||
== | === RC2 Build 2 === | ||
=== Build 1 | * [[Releases/Firefox 4RC2/Test Plan:Smoketests | Automated Smoketests]] - ashughes: '''PASS''' | ||
* [[Releases/Firefox 4RC/Test Plan:Smoketests | Smoketests and BFTs]]: ''' | * [[Releases/Firefox 4RC2/Test Plan:Updates | Automated Updates]] - ashughes: '''results''' | ||
* [[Releases/Firefox 4RC/Test Plan:Feature Area Testing | Feature Area Tests]]: ''' | * [[Releases/Firefox 4RC2/Test Plan:Focus | Specific plan for RC2]] - juanb/abillings: '''Failed''' | ||
* [[Releases/Firefox 4RC/Test Plan:Updates | Updates Checks]] - ashughes: ''' | ** Notes: Reopened bug 641630, which was not the critical bug. | ||
* [[Releases/Firefox 4RC/Test Plan:l10n | l10n Spot Checks]]: ''' | ** Limited scope: Bug verifications and testing around fixes. | ||
* [[Releases/Firefox 4RC/Test Plan:Accessibility | Accessibility]] - | |||
* [[Releases/Firefox 4RC/Test Plan:Topsites | Top Sites Testing]]: ''' | === RC1 Build 1 === | ||
* [[Releases/Firefox 4RC/Test Plan:Add-ons | Top Add-ons]] - | * [[Releases/Firefox 4RC/Test Plan:Smoketests | Smoketests and BFTs]] - automation/whimboo, ashughes, manual/Waverley : '''Completed''' | ||
* [[Releases/Firefox 4RC/Test Plan:Testday | | * [[Releases/Firefox 4RC/Test Plan:Feature Area Testing | Feature Area Tests]] - manual/Waverley, QA Team : '''Completed''' | ||
* [[Releases/Firefox 4RC/Test Plan:Updates | Updates Checks]] - automation/ashughes: '''BETATEST, RELEASETEST, BETA complete''' | |||
* [[Releases/Firefox 4RC/Test Plan:l10n | l10n Spot Checks]] - automation/whimboo: '''Completed''' | |||
* [[Releases/Firefox 4RC/Test Plan:Accessibility | Accessibility]] - manual/automated/MarcoZ: '''Signed off''' | |||
* [[Releases/Firefox 4RC/Test Plan:Topsites | Top Sites Testing]] - automation/Tomcat: '''Completed''' | |||
* [[Releases/Firefox 4RC/Test Plan:Add-ons | Top Add-ons]] - manual/QA team, testday: '''Completed''' | |||
* [[Releases/Firefox 4RC/Test Plan:Testday | Testdays]]: '''Done''' | |||
** Friday 4th - [https://quality.mozilla.org/events/2011/02/28/add-ons-compatibility-and-add-ons-manager-testday-march-4th/ Add-ons compatibility and add-ons manager] | |||
** Friday 11th - [https://quality.mozilla.org/events/2011/03/04/unconfirmed-bugs-testday-friday-march-11th/ Unconfirmed bug triage] | |||
* [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=whiteboard%3A4RC Bugs Filed]: '''crashers, blocking bugs, others''' | * [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=whiteboard%3A4RC Bugs Filed]: '''crashers, blocking bugs, others''' | ||
** Whiteboard entry '''[4rc]''' | ** Whiteboard entry '''[4rc]''' | ||
* [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=whiteboard% | * [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=whiteboard%3Aaddons-testday Bugs Filed During Testday]: '''crashers, blocking bugs, others''' | ||
** Whiteboard entry on testday: | ** Whiteboard entry on testday: [addons-testday] | ||
Latest revision as of 18:22, 30 March 2011
Firefox 4 Release Candidate Testplan
Coordinating QA: juanb in irc.mozilla.org, jbecerra@mozilla.com, mevans@mozilla.com, tchung@mozilla.com, or abillings@mozilla.com.
Description of Release
This is the first release candidate of Firefox 4. Unless serious issues are found during in-house testing or its cycle of beta testing, this will be what we ship to users. Historically, we have not shipped the first release candidate.
Test Plan for RC
As in previous milestones, we are going to run our automated tests suites, smoketests, basic functional tests, and updates tests on all shipped locales, using Mozmill, and we'll complement those with manual checks on en-US builds. In addition to the regular basic cycle of testing we'll focus on the following activities, some of which we have already done on previous betas, and some of which are specific to the release candidate:
- We are going to verify bug fixes since beta 12 and test around them with exploratory testing- Waverley/Smartware contractors - Done
- Plugins testing across platforms: Flash, Silverlight, Quicktime, Adobe Reader (latest versions), Java, RealPlayer, Shockwave, and Windows Media Player (From the list of most popular plugins in https://addons.mozilla.org/) - Waverley - Done
- We should spot check Netflix - see marcia for testing account - Done
- Web Compatibility Testing (Round 3): Popular social, news, games, and entertainment sites (See testing matrix) - uTest - Done
- Hardware compatibility testing including blocklisting: Performed by outside vendor (See on-going) - iBeta Done
- Top 50 add-ons testing: basic functionality - Done
- A round of testing Zynga games through Facebook: Mafia Wars, FarmVille, FrontierVille, using accounts provided by Zynga - marcia, CantinaVille - Done
- Accessibility - MarcoZ - Signed off
- A cycle of top sites page load testing: top 20k sites - Tomcat - Done
- Special areas sign-off: Sync (desktop and mobile) - Tracy/Tchung, Test Pilot/Feedback - juanb Signed off'
- Crash-stats stability assessment (after 3 full week days) - marcia Pending results
- Major update and downgrade scenario assessment - Waverley/Smartware/juanb - Done
- Fx4 in-product web pages sign-off - marcia, juanb
- First Run, What's New, Release Notes, Getting Started, Snippet Server (about:home)
- Web of Wonder Demos - Spotcheck for crashes and CPU spikes/usage - marcia, juanb - Done
- Web properties sign-off: AMO, SUMO, Socorro, Input - WebQA
- An RC-specific testday or possible meetup - QA team - Done
- Test coverage of quit dialog scenarios - ashughes, need others to help, zpao on hand to answer questions
During the time the RC is in the hand of users we'll be monitoring crash stats as well as feedback looking for regressions that may prompt another release candidate. We'll work closely with the support team to identify any new issues and raise them to the attention of the release drivers should they require another RC.
Any Further RCs
Should there be a need for another release candidate, we are not going to go through all of the above activities on the follow-up RC. We'll devote our time verifying the bug fixes and testing around them depending on the types of issues found. We are going to rely on new unconfirmed bugs and user feedback to assess the stability of RC. We expect the scope of issues, whether they be regressions or new problems, to be very narrow.
Schedule
When it's ready.
Builds
Test Results
RC2 Build 3 Final (Current)
- Automated Smoketests - ashughes: PASS
- Automated Updates - ashughes: PASS
- Major Update Smoketests - ashughes: PASS
- Specific plan for RC2 - team: result
- Limited scope: Bug verifications and testing around fixes.
- Advertised Major Update - ashughes/waverley: delayed
- bug 646054 - Empty update dialog for 3.6 advertised updates only
RC2 Build 2
- Automated Smoketests - ashughes: PASS
- Automated Updates - ashughes: results
- Specific plan for RC2 - juanb/abillings: Failed
- Notes: Reopened bug 641630, which was not the critical bug.
- Limited scope: Bug verifications and testing around fixes.
RC1 Build 1
- Smoketests and BFTs - automation/whimboo, ashughes, manual/Waverley : Completed
- Feature Area Tests - manual/Waverley, QA Team : Completed
- Updates Checks - automation/ashughes: BETATEST, RELEASETEST, BETA complete
- l10n Spot Checks - automation/whimboo: Completed
- Accessibility - manual/automated/MarcoZ: Signed off
- Top Sites Testing - automation/Tomcat: Completed
- Top Add-ons - manual/QA team, testday: Completed
- Testdays: Done
- Friday 4th - Add-ons compatibility and add-ons manager
- Friday 11th - Unconfirmed bug triage
- Bugs Filed: crashers, blocking bugs, others
- Whiteboard entry [4rc]
- Bugs Filed During Testday: crashers, blocking bugs, others
- Whiteboard entry on testday: [addons-testday]