QA/Firefox Updates: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
(Created page with "Tracking Page for QA involvement in testing of Firefox Updates == Silent Updates == {| |- | '''Summary:''' | Make the update pro...")
 
 
(40 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Tracking Page for QA involvement in testing of Firefox Updates
= Summary =
The following is a tracking page for QA coverage of Firefox Updates. Any and all questions should be directed to the QA Lead on this project, '''[mailto:ahughes@mozilla.com Anthony Hughes]''' (irc: ashughes).
 
== Context ==
With the move to rapid releases, QA made a risk assessment to slip on the 6 week schedule or to rely heavily on automation as a testing strategy. With that, we quickly moved from 100% manual testing to 100% automated testing. The purpose of this document is to build out and communicate a testing strategy which is inclusive of many types of testing without sacrificing time-to-market. Thereby ensuring our users the most secure and painless update experience.
 
; Current Testing Strategy
* Automation test partial, complete and fallback update paths
* Parsing automated results for any red-flags, but no focused manual spotchecking of updates
* Random slice of 4 previous versions and 4 locales across 9 platforms
* Time to sign-off updates ~1 hour per channel (4 hours from betatest to release)
 
; Revised Strategy Going Forward
* Ownership: Update component and new Update features to increase visibility into issues so we can react more aggressively
* Automation: test partial, complete and fallback update paths; parse automated results for any red-flags
** Coverage: 4 previous versions, 4 locales, 9 platforms, releasetest/beta/release
** Time to Sign-off: ~1 hour per channel
* Manual: run Smoketest suite and exploratory testing (billboard, what's new pages, places/profile/add-ons/session retention)
** Coverage: 2 previous versions, 2 locales, 4 platforms, betatest
** Time to Sign-off: ~4 hours
* This strategy should not significantly impact time to market as manual testing can happen in advance of and parallel to automation
* Total time to sign-off: ~6 hours from betatest to release
 
; Long-term Vision
* Running automation across ALL locales
* Expand automation to capture potential UX regressions
* Evolve manual testing in anticipation of and reaction to issues
* Increase QA visibility and aggressiveness to foster a proactive strategy
 
== [[QA/Firefox_Updates/General|Firefox Update Mechanism]] ==
The following will be used to track how we test Firefox updates in general.
* Litmus Tests
* Automated Tests
* Bugzilla Triage
* etc...
 
{| class="querytable"
|-
| class="header" | Summary
| Ensure a certain level of quality with the update experience in general
|-
| class="header" | QA Lead
| [mailto:ahughes@mozilla.com Anthony Hughes] (Desktop), [mailto:kbrosnan@mozilla.com Kevin Brosnan] (Mobile)
|-
| class="header" | Manager
| [mailto:rstrong@mozilla.com Robert Strong]
|-
| class="header" | QA Status
| Currently cataloging existing test coverage
|}


== [[Program_Management/Programs/Silent_Update|Silent Updates]] ==
== [[Program_Management/Programs/Silent_Update|Silent Updates]] ==
{|
{| class="querytable"
  |-
  |-
  | '''Summary:'''
  | class="header" | Summary
  | Make the update process silent so that there is the minimal user interaction required.
  | Make the update process silent so that there is the minimal user interaction required.
  |-
  |-
  | '''Stage:'''
  | class="header" | Status
  | Definition
  | Most features are in the process of landing or are in development (2011-11-22)
  |-
  |-
  | '''QA Lead:'''
  | class="header" | QA Lead
  | [mailto:ahughes@mozilla.com Anthony Hughes]
  | [mailto:ahughes@mozilla.com Anthony Hughes]
  |-
  |-
  | '''Manager:'''
  | class="header" | Manager
  | [mailto:clee@mozilla.com Chris Lee]
  | [mailto:clee@mozilla.com Chris Lee]
|-
| class="header" | QA Status
| QA is drafting test plans and test cases; testing landed features (2011-11-22)
|-
| class="header" | Meetings
| Every [[Program_Management/Programs/Silent_Update#Weekly_Meetings|Thursday]] at 1pm PDT
  |}
  |}
''This is an aggregate of several smaller features...(see below)''
''This is an aggregate of several smaller features...(see below)''


{|
{| class="querytable"
  | '''Feature'''
  | class="header" | Feature
  | '''Owner'''
  | class="header" | Owner
  | '''Test Plan'''
  | class="header" | Test Plan
  | '''Status'''
  | class="header" | Status
| class="header" | Milestone
|-
| [[Silent_Update_OS_Dialogs|Bypass UAC Dialog on Windows]]
| Simona Badau
| [[Silent Update OS Dialogs/TestPlan|test plan]]
| Test Planning
| Firefox 12
|-
| [[Silent_Update_updater|Background Updates]]
| Vlad Ghetiu
| [[Silent Update updater/TestPlan|test plan]]
| Test Planning
| Firefox 13
  |-
  |-
  | [[Silent_Update_not_now_prompt|Workflow with Incompatible Add-ons on Update]]
  | [[Silent_Update_not_now_prompt|Workflow with Incompatible Add-ons on Update]]
  | ???
  | Ioana Budnar
  | ???
  | [[Silent Update not now prompt/TestPlan|test plan]]
  | Needs Owner
| n/a
  | Unscoped
  |-
  |-
  | [[Silent_Update_updater|Apply Update on Shutdown]]
  | [[Features/Add-ons/Add-ons_Default_to_Compatible|Add-ons Compatible by Default]]
  | ???
  | Virgil Dicu
  | ???
  | [[Features/Add-ons/Add-ons Default to Compatible/TestPlan|test plan]]
  | Needs Owner
| '''SIGNED-OFF''' 2012-01-25
  | Firefox 10
  |-
  |-
  | [[Silent_Update_whatsnew|What's New Communication]]
  | [[Silent_Update_whatsnew|What's New Communication]]
  | ???
  | Mihaela Velimiroviciu
  | ???
  | [[Features/Desktop/Silent Update whatsnew/TestPlan|test plan]]
  | Needs Owner
| '''SIGNED-OFF''' 2011-11-05
  | Firefox 8
  |-
  |-
  | [[Silent_Update_OS_Dialogs|Bypass UAC Dialog on Windows]]
  | [[Firefox/Features/Lessen_App_Update_Displayed_UI|Reduce Displayed UI]]
  | ???
| Paul Silaghi
  | ???
| [[Firefox/Features/Lessen App Update Displayed UI/TestPlan|test plan]]
  | Needs Owner
| '''SIGNED-OFF''' 2011-11-05
| Firefox 8
|}
''Current overview of compatibility can be found [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdGs2NFZ5b1RST3FNWHlzVUdtTEFHUWc&hl=en_US#gid=0 here]''<br>
''Draft test cases can be found [https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdDc4emdxOE8xamJiV1ZZSEFUV0dnNnc#gid=5 here]''<br>
''Litmus test cases can be found [https://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=44&testgroup_id=573&subgroup_id= here]''<br>
''Etherpad for staging weekly status can be found [https://etherpad.mozilla.org/silent-update-status here]''
 
=== Meeting Notes ===
{| class="querytable"
|-
| class="header" style="width:10%" | Date
| class="header" style="width:30%" | Summary
| class="header" style="width:60%" | Notes
|-
| 2012-01-26
| [[Program_Management/Programs/Silent_Update/2012-01-26|Silent Update Status]]
|
UAC Service (Simona)
* Will not uplift to Firefox 11
|-
| 2012-01-05
| [[Program_Management/Programs/Silent_Update/2012-01-05|Silent Update Status]]
|
UAC Service (Simona)
* Landed on Nightly, needs testing to elevate to Aurora in time for Firefox 11
Default Compatible Add-ons (Virgil)
* Working on regression hunting test plan for real-world ecosystem
Background Updates (VladG)
* Security review scheduled for next Monday, patches landing very soon
* Ehsan will ask if we can have automatic Nightlies generated for testing
* Ehsan will try to make himself more available during Vlad's timings
|-
| 2011-12-08
| [[Program_Management/Programs/Silent_Update/2011-12-08|Silent Update Status]]
|
UAC Dialog Suppression (Simona)
* lots of backend issues plaguing landing/testing
* hopefully test ready by Monday, landing on Tuesday in Nightly
* ehsan has a tool for QA to use (he will email)
* anyone available to help should assist Simona (ie. owners for features slated for Fx 12)
Add-ons Compatible by Default (Virgil)
* Be aware of [[Features/Add-ons/Add-ons_Default_to_Compatible/Detection_and_Mitigation|Detection & Mitigation doc]]
* Focus on Dark Matter add-ons
Background Updates (Vlad)
* elm branch should be updated on elm branch
* ready by early next week
Incompatible Add-ons Workflow (Ioana)
* landing probably in january
 
|-
| 2011-12-01
| [[Program_Management/Programs/Silent_Update/2011-12-01|Silent Update Status]]
| UAC dialog suppression
* Targeting a ship of Dec 6th in Nightly
* Testing needs to begin NOW with special attention to fallback updates
* Once Nightly builds have it, we need to get the community testing it
|-
| 2011-11-24
| [[Program_Management/Programs/Silent_Update/2011-11-24|Silent Update Status]]
|
* Releng is working on bits for invalid certificated testing for MARs
* Brian to spearhead testing once ready (Dec 6 deadline)
|-
| 2011-11-17
| [[Program_Management/Programs/Silent_Update/2011-11-17|Silent Update Status]]
|
* OS Security Dialog Removal will be restricted to Windows XP, Vista, and 7
* Background Updates needs more focused testing, in spite of automation coverage
* QA interaction has been markedly improved over recent weeks
|-
| 2011-11-10
| [[Program_Management/Programs/Silent_Update/2011-11-10|Silent Update Status]]
|
* Background Updates, UAC Dialog Suppression, and Add-ons Compatible on Default are ready for testing
* escalated to QA Owners
|-
| 2011-10-21
| [[Security/Reviews/Firefox10/SilentUpdateEliminateWait|Security Review: Background Updates]]
|
* QA needs a comprehensive testing strategy for "noisy" updates which can be reused for "silent" updates
* Fallback from silent to noisy also needs coverage.
|-
| 2011-10-20
| [[Program_Management/Programs/Silent_Update/2011-10-20|Silent Update Status]]
|
* Need testplans ASAP for Apply Update on Shutdown, Bypass UAC, Default Compatible Add-ons
* What's New Page pulled from Firefox 8, exists for Firefox 9
* Testday next Friday, Oct 28 -- please add tests to the [https://etherpad.mozilla.org/testday-20111028 testplan]
|-
| 2011-10-13
| [[Program_Management/Programs/Silent_Update/2011-10-13|Silent Update Status]]
  |  
* Nothing actionable for QA
|-
| 2011-10-13
| [[Security/Reviews/Firefox10/SilentUpdateOSDialogs|Security Review: OS Dialogs]]
|
* Need testing of UAC; service on 32-bit, 64-bit, 32-on-64-bit; fallback to updater.exe on service fail
* Targetting Firefox 10
|}
 
== Add-ons ==
{| class="querytable"
  | class="header" | Feature
  | class="header" | Owner
| class="header" | Test Plan
| class="header" | Status
| class="header" | Milestone
|-
| [[Extension_Manager:Projects:Confirm_on_Upgrade|Add-ons Confirm on Upgrade]]
| Vlad Maniac
| [[Extension_Manager/QA/Feature_Test_Plans/Extension_Manager:Projects:Confirm_on_Upgrade|test plan]]
| '''SIGNED-OFF'''
| Firefox 8
  |-
  |-
  | [[Firefox/Features/Lessen_App_Update_Displayed_UI|Reduce Displayed UI]]
  | [[Extension_Manager:Projects:Third_Party_Add-on_Warnings|Third Party Add-on Warning]]
  | ???
| Vlad Maniac
  | ???
| [[Extension_Manager/QA/Feature_Test_Plans/Extension_Manager:Projects:Third_Party_Add-on_Warnings|test plan]]
  | Needs Owner
  | '''SIGNED-OFF'''
| Firefox 8
|}
 
== Mobile ==
{| class="querytable"
  | class="header" | Feature
  | class="header" | Owner
| class="header" | Test Plan
| class="header" | Status
| class="header" | Milestone
  |-
  |-
  | [[Features/Add-ons/Add-ons_Default_to_Compatible|Add-ons Compatible by Default]]
  |  
  | ???
|  
  | ???
  |  
  | Needs Owner
  |  
  |  
  |}
  |}

Latest revision as of 21:45, 26 January 2012

Summary

The following is a tracking page for QA coverage of Firefox Updates. Any and all questions should be directed to the QA Lead on this project, Anthony Hughes (irc: ashughes).

Context

With the move to rapid releases, QA made a risk assessment to slip on the 6 week schedule or to rely heavily on automation as a testing strategy. With that, we quickly moved from 100% manual testing to 100% automated testing. The purpose of this document is to build out and communicate a testing strategy which is inclusive of many types of testing without sacrificing time-to-market. Thereby ensuring our users the most secure and painless update experience.

Current Testing Strategy
  • Automation test partial, complete and fallback update paths
  • Parsing automated results for any red-flags, but no focused manual spotchecking of updates
  • Random slice of 4 previous versions and 4 locales across 9 platforms
  • Time to sign-off updates ~1 hour per channel (4 hours from betatest to release)
Revised Strategy Going Forward
  • Ownership: Update component and new Update features to increase visibility into issues so we can react more aggressively
  • Automation: test partial, complete and fallback update paths; parse automated results for any red-flags
    • Coverage: 4 previous versions, 4 locales, 9 platforms, releasetest/beta/release
    • Time to Sign-off: ~1 hour per channel
  • Manual: run Smoketest suite and exploratory testing (billboard, what's new pages, places/profile/add-ons/session retention)
    • Coverage: 2 previous versions, 2 locales, 4 platforms, betatest
    • Time to Sign-off: ~4 hours
  • This strategy should not significantly impact time to market as manual testing can happen in advance of and parallel to automation
  • Total time to sign-off: ~6 hours from betatest to release
Long-term Vision
  • Running automation across ALL locales
  • Expand automation to capture potential UX regressions
  • Evolve manual testing in anticipation of and reaction to issues
  • Increase QA visibility and aggressiveness to foster a proactive strategy

Firefox Update Mechanism

The following will be used to track how we test Firefox updates in general.

  • Litmus Tests
  • Automated Tests
  • Bugzilla Triage
  • etc...
Summary Ensure a certain level of quality with the update experience in general
QA Lead Anthony Hughes (Desktop), Kevin Brosnan (Mobile)
Manager Robert Strong
QA Status Currently cataloging existing test coverage

Silent Updates

Summary Make the update process silent so that there is the minimal user interaction required.
Status Most features are in the process of landing or are in development (2011-11-22)
QA Lead Anthony Hughes
Manager Chris Lee
QA Status QA is drafting test plans and test cases; testing landed features (2011-11-22)
Meetings Every Thursday at 1pm PDT

This is an aggregate of several smaller features...(see below)

Feature Owner Test Plan Status Milestone
Bypass UAC Dialog on Windows Simona Badau test plan Test Planning Firefox 12
Background Updates Vlad Ghetiu test plan Test Planning Firefox 13
Workflow with Incompatible Add-ons on Update Ioana Budnar test plan n/a Unscoped
Add-ons Compatible by Default Virgil Dicu test plan SIGNED-OFF 2012-01-25 Firefox 10
What's New Communication Mihaela Velimiroviciu test plan SIGNED-OFF 2011-11-05 Firefox 8
Reduce Displayed UI Paul Silaghi test plan SIGNED-OFF 2011-11-05 Firefox 8

Current overview of compatibility can be found here
Draft test cases can be found here
Litmus test cases can be found here
Etherpad for staging weekly status can be found here

Meeting Notes

Date Summary Notes
2012-01-26 Silent Update Status

UAC Service (Simona)

  • Will not uplift to Firefox 11
2012-01-05 Silent Update Status

UAC Service (Simona)

  • Landed on Nightly, needs testing to elevate to Aurora in time for Firefox 11

Default Compatible Add-ons (Virgil)

  • Working on regression hunting test plan for real-world ecosystem

Background Updates (VladG)

  • Security review scheduled for next Monday, patches landing very soon
  • Ehsan will ask if we can have automatic Nightlies generated for testing
  • Ehsan will try to make himself more available during Vlad's timings
2011-12-08 Silent Update Status

UAC Dialog Suppression (Simona)

  • lots of backend issues plaguing landing/testing
  • hopefully test ready by Monday, landing on Tuesday in Nightly
  • ehsan has a tool for QA to use (he will email)
  • anyone available to help should assist Simona (ie. owners for features slated for Fx 12)

Add-ons Compatible by Default (Virgil)

Background Updates (Vlad)

  • elm branch should be updated on elm branch
  • ready by early next week

Incompatible Add-ons Workflow (Ioana)

  • landing probably in january
2011-12-01 Silent Update Status UAC dialog suppression
  • Targeting a ship of Dec 6th in Nightly
  • Testing needs to begin NOW with special attention to fallback updates
  • Once Nightly builds have it, we need to get the community testing it
2011-11-24 Silent Update Status
  • Releng is working on bits for invalid certificated testing for MARs
  • Brian to spearhead testing once ready (Dec 6 deadline)
2011-11-17 Silent Update Status
  • OS Security Dialog Removal will be restricted to Windows XP, Vista, and 7
  • Background Updates needs more focused testing, in spite of automation coverage
  • QA interaction has been markedly improved over recent weeks
2011-11-10 Silent Update Status
  • Background Updates, UAC Dialog Suppression, and Add-ons Compatible on Default are ready for testing
  • escalated to QA Owners
2011-10-21 Security Review: Background Updates
  • QA needs a comprehensive testing strategy for "noisy" updates which can be reused for "silent" updates
  • Fallback from silent to noisy also needs coverage.
2011-10-20 Silent Update Status
  • Need testplans ASAP for Apply Update on Shutdown, Bypass UAC, Default Compatible Add-ons
  • What's New Page pulled from Firefox 8, exists for Firefox 9
  • Testday next Friday, Oct 28 -- please add tests to the testplan
2011-10-13 Silent Update Status
  • Nothing actionable for QA
2011-10-13 Security Review: OS Dialogs
  • Need testing of UAC; service on 32-bit, 64-bit, 32-on-64-bit; fallback to updater.exe on service fail
  • Targetting Firefox 10

Add-ons

Feature Owner Test Plan Status Milestone
Add-ons Confirm on Upgrade Vlad Maniac test plan SIGNED-OFF Firefox 8
Third Party Add-on Warning Vlad Maniac test plan SIGNED-OFF Firefox 8

Mobile

Feature Owner Test Plan Status Milestone
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Firefox_Updates&oldid=391017"