QA/Firefox Updates
< QA
Jump to navigation
Jump to search
Summary
The following is a tracking page for QA coverage of Firefox Updates. Any and all questions should be directed to the QA Lead on this project, Anthony Hughes (irc: ashughes).
Context
With the move to rapid releases, QA made a risk assessment to slip on the 6 week schedule or to rely heavily on automation as a testing strategy. With that, we quickly moved from 100% manual testing to 100% automated testing. The purpose of this document is to build out and communicate a testing strategy which is inclusive of many types of testing without sacrificing time-to-market. Thereby ensuring our users the most secure and painless update experience.
- Current Testing Strategy
- Automation test partial, complete and fallback update paths
- Parsing automated results for any red-flags, but no focused manual spotchecking of updates
- Random slice of 4 previous versions and 4 locales across 9 platforms
- Time to sign-off updates ~1 hour per channel (4 hours from betatest to release)
- Revised Strategy Going Forward
- Ownership: Update component and new Update features to increase visibility into issues so we can react more aggressively
- Automation: test partial, complete and fallback update paths; parse automated results for any red-flags
- Coverage: 4 previous versions, 4 locales, 9 platforms, releasetest/beta/release
- Time to Sign-off: ~1 hour per channel
- Manual: run Smoketest suite and exploratory testing (billboard, what's new pages, places/profile/add-ons/session retention)
- Coverage: 2 previous versions, 2 locales, 4 platforms, betatest
- Time to Sign-off: ~4 hours
- This strategy should not significantly impact time to market as manual testing can happen in advance of and parallel to automation
- Total time to sign-off: ~6 hours from betatest to release
- Long-term Vision
- Running automation across ALL locales
- Expand automation to capture potential UX regressions
- Evolve manual testing in anticipation of and reaction to issues
- Increase QA visibility and aggressiveness to foster a proactive strategy
Firefox Update Mechanism
The following will be used to track how we test Firefox updates in general.
- Litmus Tests
- Automated Tests
- Bugzilla Triage
- etc...
| Summary | Ensure a certain level of quality with the update experience in general |
| QA Lead | Anthony Hughes (Desktop), Kevin Brosnan (Mobile) |
| Manager | Robert Strong |
| QA Status | Currently cataloging existing test coverage |
Silent Updates
| Summary | Make the update process silent so that there is the minimal user interaction required. |
| Status | Currently planning & defining features |
| QA Lead | Anthony Hughes |
| Manager | Chris Lee |
| QA Status | Currently seeking QA owners; email sent to Softvision to solicit owners immediately (2011-10-11) |
| Meetings | Every Thursday at 1pm PDT |
This is an aggregate of several smaller features...(see below)
| Feature | Owner | Test Plan | Status | Milestone |
| Background Updates | Anthony Hughes | test plan | Test Planning | |
| Workflow with Incompatible Add-ons on Update | Ioana Budnar | test plan | Test Planning | |
| Apply Update on Shutdown | Vlad Ghetiu | test plan | Test Planning | |
| What's New Communication | Mihaela Velimiroviciu | test plan | Test Planning | Firefox 8 |
| Bypass UAC Dialog on Windows | Simona Badau | test plan | Test Planning | Firefox 10 |
| Reduce Displayed UI | Paul Silaghi | test plan | Test Planning | Firefox 10 |
| Add-ons Compatible by Default | Virgil Dicu | test plan | Test Planning | Firefox 10 |
Test cases can be found here
Meeting Notes
| Date | Summary | Notes |
| 2011-10-21 | Security Review: Background Updates | QA needs a comprehensive testing strategy for "noisy" updates which can be reused for "silent" updates Fallback from silent to noisy also needs coverage. |
| 2011-10-20 | Silent Update Status | Need testplans ASAP for Apply Update on Shutdown, Bypass UAC, Default Compatible Add-ons What's New Page pulled from Firefox 8, exists for Firefox 9 Testday next Friday, Oct 28 -- please add tests to the testplan |
| 2011-10-13 | Silent Update Status | Nothing actionable for QA |
| 2011-10-13 | Security Review: OS Dialogs | Need testing of UAC; service on 32-bit, 64-bit, 32-on-64-bit; fallback to updater.exe on service fail Targetting Firefox 10 |
Add-ons
| Feature | Owner | Test Plan | Status | Milestone |
| Add-ons Confirm on Upgrade | Vlad Maniac | test plan | Testing | Firefox 8 |
| Third Party Add-on Warning | Vlad Maniac | test plan | Testing | Firefox 8 |
Mobile
| Feature | Owner | Test Plan | Status | Milestone |