MozillaWiki

QA/Goals/2015q1: Difference between revisions

< QA‎ | Goals
 
(41 intermediate revisions by 9 users not shown)
Line 5: Line 5:
| style="font-weight: bold; background: #DDD; width: 10%" | Milestone
| style="font-weight: bold; background: #DDD; width: 10%" | Milestone
| style="font-weight: bold; background: #DDD; width: 8%" | Date
| style="font-weight: bold; background: #DDD; width: 8%" | Date
| style="font-weight: bold; background: #DDD; width: 8%" | Delight
| style="font-weight: bold; background: #DDD; width: 8%" | Deliberate
| style="font-weight: bold; background: #DDD; width: 8%" | Dependability
| style="font-weight: bold; background: #DDD; width: 8%" | Impact
| style="font-weight: bold; background: #DDD; width: 8%" | Security & Privacy
| style="font-weight: bold; background: #DDD; width: 8%" | Mozfactor
| style="font-weight: bold; background: #DDD; width: 8%" | Performance
| style="font-weight: bold; background: #DDD; width: 8%" | Overall Quality
| style="font-weight: bold; background: #DDD; width: 10%" | Overall Score
| style="font-weight: bold; background: #DDD; width: 48%" | Rationale
| style="font-weight: bold; background: #DDD; width: 40%" | Rationale
|-
|-
| .
| Firefox 35
| .
| Jan 12 2015
|
|
|
|
|
Line 21: Line 19:
|
|
|-
|-
| .
| Firefox 36
| .
| Feb 23 2015
|
|
|
|
|
Line 30: Line 27:
|
|
|-
|-
| .
| Spring Release Planning
| .
| Ongoing
|
|
|
|
|
Line 39: Line 35:
|
|
|-
|-
| .
| New Embedded Team Sprints
| .
| ongoing
|
|
|
|
|
Line 48: Line 43:
|
|
|-
|-
| .
| Marionette Training and Test Conversion
| .
| Feb 1 2015
|
|
|
|
Line 55: Line 50:
|
|
|
|
|
|-
| .
| .
|
|
|
|
|
|
|-
| .
| .
|
|
|
|
|
|
|-
| .
| .
|
|
|
|
|
|
|-
| .
| .
|
|
|
|
|
|
|-
| .
| .
|
|
|
|
|
|
|-
| .
| .
|
|
|
|
|
|
|-
|}
|}
== Prioritized Tasks ==
* [ON TRACK] [mbrandt/bsilverberg] - Identify and deliver on the highest priority/biggest pain-points of MozTrap after the API work (API to be covered by camd)
** Identified in https://www.pivotaltracker.com/n/projects/280483


== General Quality ==
== General Quality ==
* {{ok|<small>[whimboo, with help: ctalbert, kairo, releng]</small> Automate [https://github.com/mozilla/mozmill-ci/issues/535 kicking off update tests] so that the release process is automated end to end except manual spot checks}}
* {{defer|<small>[whimboo, with help: ctalbert, kairo, releng]</small> Automate [https://github.com/mozilla/mozmill-ci/issues/535 kicking off update tests] so that the release process is automated end to end except manual spot checks}}
* {{ok|<small>[stephend, joint with a*team]</small> Coordinate training and prioritize efforts to convert P1 tests from Mozmill to the Marionette "Green-light" test framework in preparation for E10s}}
** Defered due to time being spent on Mozmill -> Marionette Conversion and the possibility of doing update tests in buildbot. Will be coordinating with Releng
* {{done|<small>[stephend, joint with a*team]</small> Coordinate training and prioritize efforts to convert P1 tests from Mozmill to the Marionette "[https://github.com/mozilla/firefox-ui-tests Green-light]" test framework in preparation for E10s}}
** Training video was posted to Air Mozilla: https://air.mozilla.org/mozmill-to-marionette-conversion-bootcamp-training/
** Remaining conversion work in dependencies from this tracking bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=1129843 bug 1129843]
* {{ok|<small>[whimboo, a*team]</small> Get Marionette Green-light tests running in the QA CI system for at least Nightly tests}}
* {{ok|<small>[whimboo, a*team]</small> Get Marionette Green-light tests running in the QA CI system for at least Nightly tests}}
* {{ok|<small>[mschifer]</small> Create and monitor a set of risk factor metrics to understand risks in areas where we have no embedded QA}}
* {{ok|<small>[mschifer]</small> Create and monitor a set of risk factor metrics to understand risks in areas where we have no embedded QA}}
Line 122: Line 67:
== Firefox ==
== Firefox ==
=== E10s ===
=== E10s ===
* {{ok|<small>[lizzard, juanb]</small> - Establish baselines for automated test coverage, crash-stats, add-on compatibility, and memory footprint for e10s Nightly and track them throughout Q1.}}  
* {{done|<small>[lizzard, juanb]</small> - Establish baselines for automated-test coverage, crash-stats, add-on compatibility, and memory footprint for e10s Nightly and track them throughout Q1.}}
* {{ok|<small>[lizzard, juanb]</small> - Improve the automated test coverage for E10s enabled builds by turning on tests for a new platform and ensuring bugs filed for perma-fails.}}
** We did this for addons, automated tests, and crash stats. Memory footprint did not seem like a reliable metric to track at this time given the amount of work required and the amount of variability in the data.
* {{ok|<small>[lizzard, juanb]</small> - Engage community to help improve addon compatibility testing for e10s}}
* {{risk|<small>[lizzard, juanb]</small> - Improve the automated-test coverage for E10s-enabled builds by turning on tests for a new platform and ensuring bugs filed for perma-fails.}}
** This is {{bug|1122901}}. It is now in the hands of the automation team and the releng team and while we are pushing on it, it's not entirely in our control anymore.
* {{ok|<small>[lizzard, juanb]</small> - Engage community to help improve add-on compatibility testing for e10s}}
** milestones/deliverables: https://etherpad.mozilla.org/e10s-strategy-qa
** milestones/deliverables: https://etherpad.mozilla.org/e10s-strategy-qa


=== DOM ===
=== DOM ===
* {{ok|<small>[tracy]</small> - Document best practices for triage and test writing for DOM issues to pave the way for community involvement}}
* {{done|<small>[ashughes]</small> - Document best practices for triage and test writing for DOM issues to pave the way for community involvement}}
* {{ok|<small>[ashughes]</small> - Establish baseline metrics for specification coverage, test stability, and other DOM quality metrics and track them throughout Q1}}
** work is ongoing but the v1 doc can be found now on [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Helping_the_DOM_team developer.mozilla.org]
* {{ok|<small>[tracy, ashughes]</small> - Create several new automated tests/get existing web platform tests working to support the Picture tag and ServiceWorkers specifications}}
* {{done|<small>[ashughes]</small> - Establish baseline metrics for specification coverage, test stability, and other DOM quality metrics and track them throughout Q1}}
** milestones/deliverables: https://etherpad.mozilla.org/qa-dom-strategy
** work is ongoing but the v1 doc can be found now on [[QA/Platform/DOM/Metrics|wiki.mozilla.org]]
* {{risk|<small>[ashughes]</small> - Create several new automated tests/get existing Web-platform tests working to support the Picture tag and ServiceWorkers specifications}}
** this is at risk due to re-org and higher priority deliverables being added (eg. Betabreakers)


=== Security/Privacy ===
=== Security/Privacy ===
* {{ok|<small>[mwobensmith]</small> - Set up AWS instance to run large TLS regression test}}
* {{ok|<small>[mwobensmith]</small> - Set up AWS instance to run large TLS regression test}}
* {{ok|<small>[mwobensmith]</small> - Create a fast running subset of the larger TLS test and automate to run per-release (i.e. pre-beta)}}
* {{ok|<small>[mwobensmith]</small> - Create a fast-running subset of the larger TLS test and automate to run per-release (i.e. pre-beta)}}
* {{ok|<small>[kjoziwak]</small> - Monitor and create telemetry probes to gather data to help serve as an early warning system for risky changes}}
* {{ok|<small>[kjoziwak]</small> - Monitor and create telemetry probes to gather data to help serve as an early warning system for risky changes}}


=== Fennec ===
=== Fennec ===
* {{ok|<small>[aaronmt, kbrosnan]</small> - Slim down the per release smoketest by at least 25%}}
* {{ok|<small>[aaronmt, kbrosnan]</small> - Slim down the per-release smoketest by at least 25%}}
* {{ok|<small>[kbrosnan]</small> - Investigate using marionette/firefox greenlight on Fennec, create a plan and prioritized set of targets for the marionette team}}
* {{ok|<small>[kbrosnan]</small> - Investigate using Marionette/Firefox Green-light on Fennec, create a plan and prioritized set of targets for the Marionette team}}
* {{ok|<small>[aaronmt]</small> - Fix or discontinue appthwak automation}}
* {{ok|<small>[aaronmt]</small> - Fix or discontinue appthwak automation}}


== Web QA ==
== Web QA ==
* {{prev|<small>[mbrandt and shared with MDN dev]</small> Develop and stand up a JavaScript-based testing infrastructure (proof-of-concept)}} which allows Web development and Web QA to be maximally effective by sharing key points of test infrastructure, visibility, and process/workflow:
* {{done|<small>[rbillings]</small> - Promulgate the usage of One and Done beyond QA: meet with at least one other team [ex: CBT, Security] to educate them on what O&D does and how they might use it and detail out the changes that would be required to update the scope of the site beyond QA.}}
** {{prev|<small>[mbrandt]</small> at the end of the quarter, do a writeup (blog post?) covering the progress and any next steps for Q1 2015}}
* {{done|<small>[mbrandt and shared with MDN dev]</small> Develop and stand up a JavaScript-based testing infrastructure (proof-of-concept)}} which allows Web development and Web QA to be maximally effective by sharing key points of test infrastructure, visibility, and process/workflow:
* {{ok|<small>[bsilverberg?]</small> - QA Metrics Dashboard - collate metrics on a page of links as new metrics are proven to be useful}}
** {{done|<small>[mbrandt]</small> at the end of the quarter, do a writeup (blog post?) covering the progress and any next steps for Q1 2015}}
* {{ok|<small>[davehunt]</small> - bug 1112555 Make Web QA's Jenkins public}}
* {{done|<small>[bsilverberg]</small> - QA Metrics Dashboard - collate metrics on a page of links as new metrics are proven to be useful}}
** this is to support, and have a dependency on, Kairo's goal under "General Quality," above
*** Added http://arewestableyet.com/graph/ to http://bobsilverberg.github.io/qa-dashboard/
* {{done|<small>[davehunt]</small> - bug 1112555 - Make Web QA's Jenkins public}}
** https://blog.mozilla.org/webqa/2015/03/05/web-qa-publishes-automated-test-results/
* {{ok|<small>[mbrandt/bsilverberg] - </small>Identify and deliver on the highest [https://www.pivotaltracker.com/n/projects/280483 priority/biggest pain-points of MozTrap after the API work (API to be covered by camd)]}}  
* {{ok|<small>[mbrandt/bsilverberg] - </small>Identify and deliver on the highest [https://www.pivotaltracker.com/n/projects/280483 priority/biggest pain-points of MozTrap after the API work (API to be covered by camd)]}}  
* {{prev|<small>[krupa]</small> - Have automation coverage for Payments including in-app payments}}
* {{done|<small>[krupa]</small> - Have automation coverage for Payments including in-app payments}}
** total scope of work (done vs. remaining) here: [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1015251&hide_resolved=0 bug 1015251]
** total scope of work (done vs. remaining) here: [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1015251&hide_resolved=0 bug 1015251]
* {{ok|<small>[krupa]</small> - Coordinate and drive quality efforts to prepare for AMO to require addon signing}}
** In-app Payments tests landed in https://github.com/mozilla/marketplace-tests-gaia/commit/e39b0c2a0ac3061d9d511ba9010e09ceece8ac31
* {{done|<small>[krupa]</small> - Coordinate and drive quality efforts to prepare for AMO to require add-on signing}}
** Still on-track; landing in Fx39 Beta, in mid-May


== Platform QA ==
== Platform QA ==
* {{ok|<small>[Marcia, Syd]</small> Eliminate Flash issues on youtube by supporting the media team in shipping MSE on youtube for windows vista and higher through manual and automated testing}}
* {{ok|<small>[Marcia, Syd]</small> Eliminate Flash issues on YouTube by supporting the media team in shipping MSE on YouTube for Windows Vista and higher through manual and automated testing}}
 
=== Prioritized Tasks ===
* {{done|<small>[Maja]</small> Create and deploy a suite of automated YouTube playback tests. }}
* {{done|<small>[Maja]</small> Write a tool to pull a subset of YouTube URLs from crash-stats and feed that to the automated playback tests. }}


== Community ==
== Community ==
* [marcia] Build community effort to support the Buddy Up project
* {{done|<small>[marcia]</small> Build community effort to support the Buddy Up project }}
* [ashughes] Help create at least three small, focused test-day events that focus on helping contributors be more effective
* {{ok|<small>[ashughes]</small> Help create at least three small, focused test-day events that focus on helping contributors be more effective}}
** we have three events planned this quarter: Bangladesh, Argentina/Spain, and the UK

Latest revision as of 19:11, 1 April 2015

Milestones this Quarter

This is the list of milestones this quarter. The table below is used to judge how our efforts shaped each axis of quality in the final release.

Milestone Date Deliberate Impact Mozfactor Overall Quality Rationale
Firefox 35 Jan 12 2015
Firefox 36 Feb 23 2015
Spring Release Planning Ongoing
New Embedded Team Sprints ongoing
Marionette Training and Test Conversion Feb 1 2015

Prioritized Tasks

General Quality

  • [DEFER] [whimboo, with help: ctalbert, kairo, releng] Automate kicking off update tests so that the release process is automated end to end except manual spot checks
    • Defered due to time being spent on Mozmill -> Marionette Conversion and the possibility of doing update tests in buildbot. Will be coordinating with Releng
  • [DONE] [stephend, joint with a*team] Coordinate training and prioritize efforts to convert P1 tests from Mozmill to the Marionette "Green-light" test framework in preparation for E10s
  • [ON TRACK] [whimboo, a*team] Get Marionette Green-light tests running in the QA CI system for at least Nightly tests
  • [ON TRACK] [mschifer] Create and monitor a set of risk factor metrics to understand risks in areas where we have no embedded QA
  • [ON TRACK] [kairo] Create a better concept for tracking crash-stats that more reflects the actual risk and severity of the crashes involved (than the current top 10 list)

Firefox

E10s

  • [DONE] [lizzard, juanb] - Establish baselines for automated-test coverage, crash-stats, add-on compatibility, and memory footprint for e10s Nightly and track them throughout Q1.
    • We did this for addons, automated tests, and crash stats. Memory footprint did not seem like a reliable metric to track at this time given the amount of work required and the amount of variability in the data.
  • [AT RISK] [lizzard, juanb] - Improve the automated-test coverage for E10s-enabled builds by turning on tests for a new platform and ensuring bugs filed for perma-fails.
    • This is bug 1122901. It is now in the hands of the automation team and the releng team and while we are pushing on it, it's not entirely in our control anymore.
  • [ON TRACK] [lizzard, juanb] - Engage community to help improve add-on compatibility testing for e10s

DOM

  • [DONE] [ashughes] - Document best practices for triage and test writing for DOM issues to pave the way for community involvement
  • [DONE] [ashughes] - Establish baseline metrics for specification coverage, test stability, and other DOM quality metrics and track them throughout Q1
  • [AT RISK] [ashughes] - Create several new automated tests/get existing Web-platform tests working to support the Picture tag and ServiceWorkers specifications
    • this is at risk due to re-org and higher priority deliverables being added (eg. Betabreakers)

Security/Privacy

  • [ON TRACK] [mwobensmith] - Set up AWS instance to run large TLS regression test
  • [ON TRACK] [mwobensmith] - Create a fast-running subset of the larger TLS test and automate to run per-release (i.e. pre-beta)
  • [ON TRACK] [kjoziwak] - Monitor and create telemetry probes to gather data to help serve as an early warning system for risky changes

Fennec

  • [ON TRACK] [aaronmt, kbrosnan] - Slim down the per-release smoketest by at least 25%
  • [ON TRACK] [kbrosnan] - Investigate using Marionette/Firefox Green-light on Fennec, create a plan and prioritized set of targets for the Marionette team
  • [ON TRACK] [aaronmt] - Fix or discontinue appthwak automation

Web QA

Platform QA

  • [ON TRACK] [Marcia, Syd] Eliminate Flash issues on YouTube by supporting the media team in shipping MSE on YouTube for Windows Vista and higher through manual and automated testing

Prioritized Tasks

  • [DONE] [Maja] Create and deploy a suite of automated YouTube playback tests.
  • [DONE] [Maja] Write a tool to pull a subset of YouTube URLs from crash-stats and feed that to the automated playback tests.

Community

  • [DONE] [marcia] Build community effort to support the Buddy Up project
  • [ON TRACK] [ashughes] Help create at least three small, focused test-day events that focus on helping contributors be more effective
    • we have three events planned this quarter: Bangladesh, Argentina/Spain, and the UK
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Goals/2015q1&oldid=1065995"