QA/Goals/2015q1

Milestones this Quarter

This is the list of milestones this quarter. The table below is used to judge how our efforts shaped each axis of quality in the final release.

Milestone	Date	Deliberate	Impact	Mozfactor	Overall Quality	Rationale
Firefox 35	Jan 12 2015					} General Quality [ON TRACK] [whimboo, with help: ctalbert, kairo, releng] Automate kicking off update tests so that the release process is automated end to end except manual spot checks [ON TRACK] [stephend, joint with a*team] Coordinate training and prioritize efforts to convert P1 tests from Mozmill to the Marionette "Green-light" test framework in preparation for E10s [ON TRACK] [whimboo, a*team] Get Marionette Green-light tests running in the QA CI system for at least Nightly tests [ON TRACK] [mschifer] Create and monitor a set of risk factor metrics to understand risks in areas where we have no embedded QA [ON TRACK] [kairo] Create a better concept for tracking crash-stats that more reflects the actual risk and severity of the crashes involved (than the current top 10 list) Firefox E10s [ON TRACK] [lizzard, juanb] - Establish baselines for automated test coverage, crash-stats, add-on compatibility, and memory footprint for e10s Nightly and track them throughout Q1. [ON TRACK] [lizzard, juanb] - Improve the automated test coverage for E10s enabled builds by turning on tests for a new platform and ensuring bugs filed for perma-fails. [ON TRACK] [lizzard, juanb] - Engage community to help improve addon compatibility testing for e10s milestones/deliverables: https://etherpad.mozilla.org/e10s-strategy-qa DOM [ON TRACK] [tracy] - Document best practices for triage and test writing for DOM issues to pave the way for community involvement [ON TRACK] [ashughes] - Establish baseline metrics for specification coverage, test stability, and other DOM quality metrics and track them throughout Q1 [ON TRACK] [tracy, ashughes] - Create several new automated tests/get existing web platform tests working to support the Picture tag and ServiceWorkers specifications milestones/deliverables: https://etherpad.mozilla.org/qa-dom-strategy Security/Privacy [ON TRACK] [mwobensmith] - Set up AWS instance to run large TLS regression test [ON TRACK] [mwobensmith] - Create a fast running subset of the larger TLS test and automate to run per-release (i.e. pre-beta) [ON TRACK] [kjoziwak] - Monitor and create telemetry probes to gather data to help serve as an early warning system for risky changes Fennec [ON TRACK] [aaronmt, kbrosnan] - Slim down the per release smoketest by at least 25% [ON TRACK] [kbrosnan] - Investigate using marionette/firefox greenlight on Fennec, create a plan and prioritized set of targets for the marionette team [ON TRACK] [aaronmt] - Fix or discontinue appthwak automation Web QA [CARRY OVER] [mbrandt and shared with MDN dev] Develop and stand up a JavaScript-based testing infrastructure (proof-of-concept) which allows Web development and Web QA to be maximally effective by sharing key points of test infrastructure, visibility, and process/workflow: [CARRY OVER] [mbrandt] at the end of the quarter, do a writeup (blog post?) covering the progress and any next steps for Q1 2015 [ON TRACK] [bsilverberg?] - QA Metrics Dashboard - collate metrics on a page of links as new metrics are proven to be useful [ON TRACK] [davehunt] - bug 1112555 Make Web QA's Jenkins public [ON TRACK] [mbrandt/bsilverberg] - Identify and deliver on the highest priority/biggest pain-points of MozTrap after the API work (API to be covered by camd) [CARRY OVER] [krupa] - Have automation coverage for Payments including in-app payments total scope of work (done vs. remaining) here: bug 1015251 [ON TRACK] [krupa] - Coordinate and drive quality efforts to prepare for AMO to require addon signing Platform QA [ON TRACK] [Marcia, Syd] Eliminate Flash issues on youtube by supporting the media team in shipping MSE on youtube for windows vista and higher through manual and automated testing Community [marcia] Build community effort to support the Buddy Up project [ashughes] Help create at least three small, focused test-day events that focus on helping contributors be more effective