QA/Goals/Q4

Below are the Q4 2011 Goals for the Mozilla QA organization.

Toplevel QA Organization Goals

Below are the top level goals to be achieved by the QA organization for Q4.

Maximize Desktop Firefox

• [NEW] Fully automate Firefox update testing under a continuous integration system that allows online and on-demand initiation of update testing jobs.
• [NEW] Define and execute focused quality plan to ensure Desktop Firefox upgrades have the expected minimal impact to users and add-on developers.

Maxmize Mobile Firefox

• [NEW] Drive the creation of the quality plan and initial testing to support the java Fennec UI release.
• [NEW] Creation of a prototype of a Fennec driver for usage with Selenium and Mozmill to support Fennec test automation.

Establish Credible Apps and Identity Systems

• [NEW] Drive the overall testing and quality plan to support the various components and services of the Open Web Apps release and Q4 launch.
• [NEW] Drive the quality plan and testing BrowserID for Q4 production launch and host 2 BrowserID testdays.

Advance Boot to Gecko

• [NEW] Support exploratory testing of initial releases of Boot to Gecko and define QA test strategy for future Boot to Gecko releases.

Grow Mozilla

• [NEW] Fully deploy and evangelize the QA ReMo SIG [program https://wiki.mozilla.org/ReMo/SIGs] and report experience and results to Mozilla All.
• [NEW] Develop QA Organization vision and goals for 2012 and publicly evangelize and present internally and to our community.

Desktop Firefox QA Goals

• [NEW] Timely and quality delivery of Firefox 8 and 9
• [NEW] Define and execute focused quality plan to ensure Desktop Firefox upgrades have the expected minimal impact to users and add-on developers
• [NEW] Conduct contributor path audit for Desktop Firefox QA
• [NEW] Redefine and improve bug workflow with test partners to mitigate missed issues and chemspills
• [NEW] Enable Softvision to take on more responsibility in milestone testing and explain strategy to all stakeholders
• [NEW] Involve the community more in discussions about proposals, projects, processes, and how to get involved with Desktop Firefox QA (blogs and mailing lists) and reach out to them when they reach out to us
• [NEW] Work with Automation Services to get milestone automation running in CI

see sync-pad for brainstorming notes

Browser Technology QA Goals

Mobile QA Goals

• [NEW] test support for Firefox 8 and 9
• [NEW] test support for alpha version of Native Fennec
• [NEW] Support exploratory testing of initial releases of Boot to Gecko and define QA test strategy for future Boot to Gecko releases.
• [CARRY OVER] Complete development and deploy the crowd-sourcing Firefox Mobile test harness that allows users, developers and mobile community members to run Firefox Mobile unit tests and report those results to online repository for analysis of potential platform compatibility issues
• [NEW] Complete a final draft and develop a proof-of-concept for a mobile GUI Automation framework
• [NEW] add Browser-Chrome and Mochitests to a working Crowdsource addon
  • question: can this overlap with the GUI tests?
• [NEW] Crash reporting and memshrink:
  • [NEW] At least 5 Crashing bugs reproduced
  • [NEW] Improve the skiplist
    • [NEW] 5% less meaningless crash signatures
• [NEW] need Goal around community reach.
  • started QMO and newsletter in Q3. suggest pulling the community section from last all hands and measure one

Services

• [NEW] Ship BrowserID to production, and host 2 testdays
• [NEW] Test delivered products within the Beta Services lab environment
• [NEW] Automate Sync smoketests and 6 BFTs in Mozmill and run within TPS
• [CARRY OVER] Port funkload testcases into staging and production environment
• [NEW] need Goal around community reach. suggest pulling the community section from last all hands and measure one

Labs

• [NEW] Test support for WebAPI project
• [CARRY OVER] Develop an automation framework for Pancake
• [NEW] host a pancake testday (release dependent)
• [NEW] Test Support for mobile WebApps goal on native Android app and Fennec. Still need collaboration on measuring this

WebQA Team Goals

• Automation:
  • [NEW] Convert at least the AMO BFT to use WebDriver API (P1)
  • [NEW] Abstract and prepare AMO tests for a cutover -> Open Web Apps, by removing strings "addons" and "amo" from objects and method names in AMO automation (P1)
  • [NEW] Ensure a performant AMO Marketplace/OWA store by load and perf-testing (P1)
  • [NEW] Selenium templates for Engagement campaigns/microsites (using Webdriver!) (raymond) (P2)
  • [NEW] BFT-level support for Affiliates (P2)
  • [DONE] BFT-level support for Mozillians (P2)
  • [NEW] Basic authentication tests using Selenium for BrowserID (P2)
  • Input:
    • [CARRY OVER] Prototype: move Input tests to Webdev's Jenkins instance (mbrandt) (P2)
    • [CARRY OVER] Have a set of smoketests or BFT able to run in production (mbrandt) (P2)
  • [NEW] For the next time's notes, have the link ready in the present meeting's notes (P3)
• Infrastructure:
  • [NEW] Investigate/resolve Selenium Grid session issue (P1)
  • [NEW] Integrate our builds/jobs wholesale in Webdev's Jenkins (P1)
  • [NEW] Investigate/add authentication support in Selenium (P2)
    • For dialog prompts
• Community
  • [NEW] Improve onramp for building AMO Selenium/Webdriver tests locally (marlenac) (P1)
  • [NEW] Audit our community contribution paths What are we doing well, what can we improve upon? Implement two of the recommendations by end of the quarter (P1)
  • [NEW] SUMO Community building/Forum Testdays [rbillings] (P2)
  • [NEW] Assist mluna in creating SUMO Volunteer Guidelines [rbillings] (P2)

QA Automation Services Goals

• [NEW] Implementation of a remote mechanism to trigger Mozmill functional and update tests for Firefox releases
• [NEW] Creation of a prototype of a Fennec driver for usage with Selenium and Mozmill on mobile devices
• [NEW] Enhance new members and mentors pages to actively gain and retain new contributors
• [NEW] Setup CI systems under Jenkins for daily en-US Firefox builds and for all l10n Firefox builds generated by check-ins from localizers
• [NEW] Enhance the current Selenium grid by adding real Android devices and an iOS Simulator, and proof the integration by having at least one test written for Selenium, which can be executed on all those platforms.
• [NEW] Definition of long term team goals for 2012

Case Conductor

• [NEW] Get instance running in-house with IT
• [NEW] Ship .6 for uTest Demo on Oct 20th
• [NEW] Ship 1.0 version
• [NEW] Create test cases in CC to test CC
• [NEW] Create Selenium/Lettuce tests to test CC
• [NEW] Transition at least 2 projects in-house to use Case Conductor
• [NEW] Create screencasts to show how to perform basic funcitons

QA Community Goals

• [NEW] Draft goals are here: https://etherpad.mozilla.org/QA-Q4-Goals
• QA Q4 Community Goals