QA/Goals/2015q1

Milestones this Quarter

This is the list of milestones this quarter. The table below is used to judge how our efforts shaped each axis of quality in the final release.

Milestone	Date	Deliberate	Impact	Mozfactor	Overall Quality	Rationale
Firefox 35	Jan 12 2015
Firefox 36	Feb 23 2015
Spring Release Planning	Ongoing
New Embedded Team Sprints	ongoing
Marionette Training and Test Conversion	Feb 1 2015

Prioritized Tasks

• [ON TRACK] [mbrandt/bsilverberg] - Identify and deliver on the highest priority/biggest pain-points of MozTrap after the API work (API to be covered by camd)
  • Identified in https://www.pivotaltracker.com/n/projects/280483

General Quality

• [DEFER] [whimboo, with help: ctalbert, kairo, releng] Automate kicking off update tests so that the release process is automated end to end except manual spot checks
  • Defered due to time being spent on Mozmill -> Marionette Conversion and the possibility of doing update tests in buildbot. Will be coordinating with Releng
• [DONE] [stephend, joint with a*team] Coordinate training and prioritize efforts to convert P1 tests from Mozmill to the Marionette "Green-light" test framework in preparation for E10s
  • Training video was posted to Air Mozilla: https://air.mozilla.org/mozmill-to-marionette-conversion-bootcamp-training/
  • Remaining conversion work in dependencies from this tracking bug: bug 1129843
• [ON TRACK] [whimboo, a*team] Get Marionette Green-light tests running in the QA CI system for at least Nightly tests
• [ON TRACK] [mschifer] Create and monitor a set of risk factor metrics to understand risks in areas where we have no embedded QA
• [ON TRACK] [kairo] Create a better concept for tracking crash-stats that more reflects the actual risk and severity of the crashes involved (than the current top 10 list)

Firefox

E10s

• [DONE] [lizzard, juanb] - Establish baselines for automated-test coverage, crash-stats, add-on compatibility, and memory footprint for e10s Nightly and track them throughout Q1.
  • We did this for addons, automated tests, and crash stats. Memory footprint did not seem like a reliable metric to track at this time given the amount of work required and the amount of variability in the data.
• [AT RISK] [lizzard, juanb] - Improve the automated-test coverage for E10s-enabled builds by turning on tests for a new platform and ensuring bugs filed for perma-fails.
  • This is bug 1122901. It is now in the hands of the automation team and the releng team and while we are pushing on it, it's not entirely in our control anymore.
• [ON TRACK] [lizzard, juanb] - Engage community to help improve add-on compatibility testing for e10s
  • milestones/deliverables: https://etherpad.mozilla.org/e10s-strategy-qa

DOM

• [DONE] [ashughes] - Document best practices for triage and test writing for DOM issues to pave the way for community involvement
  • work is ongoing but the v1 doc can be found now on developer.mozilla.org
• [DONE] [ashughes] - Establish baseline metrics for specification coverage, test stability, and other DOM quality metrics and track them throughout Q1
  • work is ongoing but the v1 doc can be found now on wiki.mozilla.org
• [AT RISK] [ashughes] - Create several new automated tests/get existing Web-platform tests working to support the Picture tag and ServiceWorkers specifications
  • this is at risk due to re-org and higher priority deliverables being added (eg. Betabreakers)

Security/Privacy

• [ON TRACK] [mwobensmith] - Set up AWS instance to run large TLS regression test
• [ON TRACK] [mwobensmith] - Create a fast-running subset of the larger TLS test and automate to run per-release (i.e. pre-beta)
• [ON TRACK] [kjoziwak] - Monitor and create telemetry probes to gather data to help serve as an early warning system for risky changes

Fennec

• [ON TRACK] [aaronmt, kbrosnan] - Slim down the per-release smoketest by at least 25%
• [ON TRACK] [kbrosnan] - Investigate using Marionette/Firefox Green-light on Fennec, create a plan and prioritized set of targets for the Marionette team
• [ON TRACK] [aaronmt] - Fix or discontinue appthwak automation

Web QA

• [DONE] [rbillings] - Promulgate the usage of One and Done beyond QA: meet with at least one other team [ex: CBT, Security] to educate them on what O&D does and how they might use it and detail out the changes that would be required to update the scope of the site beyond QA.
• [DONE] [mbrandt and shared with MDN dev] Develop and stand up a JavaScript-based testing infrastructure (proof-of-concept) which allows Web development and Web QA to be maximally effective by sharing key points of test infrastructure, visibility, and process/workflow:
  • [DONE] [mbrandt] at the end of the quarter, do a writeup (blog post?) covering the progress and any next steps for Q1 2015
• [DONE] [bsilverberg] - QA Metrics Dashboard - collate metrics on a page of links as new metrics are proven to be useful
  • this is to support, and have a dependency on, Kairo's goal under "General Quality," above
    • Added http://arewestableyet.com/graph/ to http://bobsilverberg.github.io/qa-dashboard/
• [DONE] [davehunt] - bug 1112555 - Make Web QA's Jenkins public
  • https://blog.mozilla.org/webqa/2015/03/05/web-qa-publishes-automated-test-results/
• [ON TRACK] [mbrandt/bsilverberg] - Identify and deliver on the highest priority/biggest pain-points of MozTrap after the API work (API to be covered by camd)
• [DONE] [krupa] - Have automation coverage for Payments including in-app payments
  • total scope of work (done vs. remaining) here: bug 1015251
  • In-app Payments tests landed in https://github.com/mozilla/marketplace-tests-gaia/commit/e39b0c2a0ac3061d9d511ba9010e09ceece8ac31
• [DONE] [krupa] - Coordinate and drive quality efforts to prepare for AMO to require add-on signing
  • Still on-track; landing in Fx39 Beta, in mid-May

Platform QA

• [ON TRACK] [Marcia, Syd] Eliminate Flash issues on YouTube by supporting the media team in shipping MSE on YouTube for Windows Vista and higher through manual and automated testing

Prioritized Tasks

• [DONE] [Maja] Create and deploy a suite of automated YouTube playback tests.
• [DONE] [Maja] Write a tool to pull a subset of YouTube URLs from crash-stats and feed that to the automated playback tests.

Community

• [DONE] [marcia] Build community effort to support the Buddy Up project
• [ON TRACK] [ashughes] Help create at least three small, focused test-day events that focus on helping contributors be more effective
  • we have three events planned this quarter: Bangladesh, Argentina/Spain, and the UK