QA/Firefox Updates: Difference between revisions
< QA
Jump to navigation
Jump to search
(Created page with "Tracking Page for QA involvement in testing of Firefox Updates == Silent Updates == {| |- | '''Summary:''' | Make the update pro...") |
No edit summary |
||
| Line 1: | Line 1: | ||
Tracking Page for QA involvement in testing of Firefox Updates | Tracking Page for QA involvement in testing of Firefox Updates | ||
== Context == | |||
With the move to rapid releases, QA made a risk assessment to slip on the 6 week schedule or to rely heavily on automation as a testing strategy. With that, we quickly moved from 100% manual testing to 100% automated testing. The purpose of this document is to build out and communicate a testing strategy which is inclusive of many types of testing without sacrificing time-to-market. Thereby ensuring our users the most secure and painless update experience. | |||
; Strategy before Rapid Release | |||
* Manually test partial, complete, and fallback update paths | |||
* Exploratory spotchecking of the update experience (billboard, what's new pages, places/profile/add-ons/session retention) | |||
* 2 previous versions and 2 locales across 4 platforms | |||
* Time to sign-off updates ~4 hours per channel (16 hours from betatest to release) | |||
; Strategy after Rapid Release | |||
* Automation test partial, complete and fallback update paths | |||
* Parsing automated results for any red-flags, but no focused manual spotchecking of updates | |||
* Random slice of 4 previous versions and 4 locales across 9 platforms | |||
* Time to sign-off updates ~1 hour per channel (4 hours from betatest to release) | |||
; Strategy Going Forward | |||
* Ownership: Update component and new Update features to increase visibility into issues so we can react more aggressively | |||
* Automation: test partial, complete and fallback update paths; parse automated results for any red-flags | |||
** Coverage: 4 previous versions, 4 locales, 9 platforms, releasetest/beta/release | |||
** Time to Sign-off: ~1 hour per channel | |||
* Manual: run Smoketest suite and exploratory testing (billboard, what's new pages, places/profile/add-ons/session retention) | |||
** Coverage: 2 previous versions, 2 locales, 4 platforms, betatest | |||
** Time to Sign-off: ~4 hours | |||
* This strategy should not significantly impact time to market as manual testing can happen in advance of and parallel to automation | |||
* Total time to sign-off: ~6 hours from betatest to release | |||
; Long-term Vision | |||
* Running automation across ALL locales | |||
* Expand automation to capture potential UX regressions | |||
* Evolve manual testing in anticipation of and reaction to issues | |||
* Increase QA visibility and aggressiveness to foster a proactive strategy | |||
== [[Program_Management/Programs/Silent_Update|Silent Updates]] == | == [[Program_Management/Programs/Silent_Update|Silent Updates]] == | ||
Revision as of 22:31, 11 October 2011
Tracking Page for QA involvement in testing of Firefox Updates
Context
With the move to rapid releases, QA made a risk assessment to slip on the 6 week schedule or to rely heavily on automation as a testing strategy. With that, we quickly moved from 100% manual testing to 100% automated testing. The purpose of this document is to build out and communicate a testing strategy which is inclusive of many types of testing without sacrificing time-to-market. Thereby ensuring our users the most secure and painless update experience.
- Strategy before Rapid Release
- Manually test partial, complete, and fallback update paths
- Exploratory spotchecking of the update experience (billboard, what's new pages, places/profile/add-ons/session retention)
- 2 previous versions and 2 locales across 4 platforms
- Time to sign-off updates ~4 hours per channel (16 hours from betatest to release)
- Strategy after Rapid Release
- Automation test partial, complete and fallback update paths
- Parsing automated results for any red-flags, but no focused manual spotchecking of updates
- Random slice of 4 previous versions and 4 locales across 9 platforms
- Time to sign-off updates ~1 hour per channel (4 hours from betatest to release)
- Strategy Going Forward
- Ownership: Update component and new Update features to increase visibility into issues so we can react more aggressively
- Automation: test partial, complete and fallback update paths; parse automated results for any red-flags
- Coverage: 4 previous versions, 4 locales, 9 platforms, releasetest/beta/release
- Time to Sign-off: ~1 hour per channel
- Manual: run Smoketest suite and exploratory testing (billboard, what's new pages, places/profile/add-ons/session retention)
- Coverage: 2 previous versions, 2 locales, 4 platforms, betatest
- Time to Sign-off: ~4 hours
- This strategy should not significantly impact time to market as manual testing can happen in advance of and parallel to automation
- Total time to sign-off: ~6 hours from betatest to release
- Long-term Vision
- Running automation across ALL locales
- Expand automation to capture potential UX regressions
- Evolve manual testing in anticipation of and reaction to issues
- Increase QA visibility and aggressiveness to foster a proactive strategy
Silent Updates
| Summary: | Make the update process silent so that there is the minimal user interaction required. |
| Stage: | Definition |
| QA Lead: | Anthony Hughes |
| Manager: | Chris Lee |
This is an aggregate of several smaller features...(see below)
| Feature | Owner | Test Plan | Status |
| Workflow with Incompatible Add-ons on Update | ??? | ??? | Needs Owner |
| Apply Update on Shutdown | ??? | ??? | Needs Owner |
| What's New Communication | ??? | ??? | Needs Owner |
| Bypass UAC Dialog on Windows | ??? | ??? | Needs Owner |
| Reduce Displayed UI | ??? | ??? | Needs Owner |
| Add-ons Compatible by Default | ??? | ??? | Needs Owner |