QA/TDAI/Goals/2009-Q2

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

This is both the agenda and the goals landing page for the Test Development and Automation team for Q2 2009. This is not official yet. It is in draft form.

High Level Goals

• Increase visibility and expand automation by completing the build integration work for JS Ref Test, Mozmill, and Maemo automation harnesses.
• Support the 3.6 feature and performance work with functional test writing and aid in detailing/specifying new performance metrics (work with Fx Front End and Platform).
• Expand our test coverage by recruiting developers and community to write targeted tests in critical areas and by devoting one day every other week to publicly work on test development for all available harnesses.
• Maintain and improve test tools to better enable people to do both manual and automated testing.
  • QAC, Mozmill, and the Results server are the tools we aim to update this quarter.

Test Automation

Focus: Improve the test automation system at harness level.

Goal: Increase visibility and expand automation by completing the build integration work for JS Ref Test, Mozmill, and Maemo automation harnesses.

Tactics:

• Institute a measuring system for tinderbox realibility
• Debug the known, observed, random failures, create a plan for handling random failures going forward.
• Land JS Ref test framework bug 469718
• Enable Build/Test infrastructure to run Mozmill tests (unit tests at least)
  • Have Mozmill tests running automatically, reporting to results server and/or staging tboxes (as appropriate)
• Integrate Maemkit with Tinderbox, hand off to build

Other Harness Level Projects

• Research and develop tools to run unittests on Windows Mobile. This includes remote launching and refactoring existing tools
  • Prototype in Q2
  • formally finished in Q3
• Create a method for writing Fennec specific automated mochitests, xpcshell tests, reftests etc.

Test Development

Focus:Expand test coverage across all harnesses and in all areas where more test coverage is needed.

Goals:

• Goal 1: Support the 3.6 feature and performance work with functional test writing and aid in detailing/specifying new performance metrics (work with Fx Front End and Platform).
• Goal 2: Expand our test coverage by recruiting developers and community to write targeted tests in critical areas and by devoting one day every other week to publicly work on test development for all available harnesses.

Tactics:

Goal 1:

• Be involved in planning discussions for our areas of focus
• Draft test priority documents detailing the new features we will concentrate on for functional testing, get at developer review of those
• Write those functional tests
• Aid Mobile team as needed with their performance testing (seems like Taras is doing much of the heavy lifting?)
• Gather requirements and create mechanism for Firefox front end team's user performance metrics from "dirty" profiles
• Gather requirements and gather platform performance metrics surrounding parsing performance for 3.5

Goal 2:

• Create a list of 'hot-spots' where we need more testing and devote 10 hours a week to writing test cases each in our specific areas. Determine the 'hot-spots' by analyzing:
  • • Query recent in-test-suite-? flags to find areas where people have flagged areas for more test development
    • Use those areas to drill down into relevant files and determine where to concentrate test cases based on areas where we have low/incomplete code coverage
    • Prioritize recently fixed security bugs for inclusion on the hot-spot list
  • Create the 'hot-spot' list for each area - JS, Content, Layout/Gfx, Mobile/Ffx Frontend
  • Bring the list to developers to get them to help create tests
  • Not having done this before, use a time based approach. Next time we will look at number of test cases written this quarter and see how to change this goal going forward

Community Leadership

Focus:Broaden the Test Development Team's reach and impact by recruiting more people to help and being more public in all we do.

Goal:Hold test development days where we publicly focus on writing tests on our 'hot-list' for each of our different test harnesses. Invite community and developers to join us.

Tactics:

• Use our 'hot-spot' list mentioned above as a resource of areas to work from when developing new tests
• Other avenues might be established by creating a well defined set of tools/wishlist projects (including detailed, concrete steps we need people to take) that we want help with: Litmus Redesign, Qac, Mozmill Helper Functions, Mozmill Features, Community Reftesting, etc.

Test Tools

Focus: Create, maintain, and improve testing tools needed for ourselves and the larger testing community

Goals:

• Provide a new QAC to the testing community. Focus on fixing usability issues with the tool so that people can more easily use that tool in their testing of Firefox.
• Code and release Mozmill 1.2, addressing issues that block some segments of testing such as A11y and restartability
  • Support the QAE team by writing Mozmill Helper Functions as needed to make test writing easier.(Full set of bugs on this unknown, expect to write between 3 and 7 helper functions over the quarter, some may actually be folded into the main product if appropriate).
  • Tailor the JS Learning Project for Mozmill test writing in order to better enable QAE to write Mozmill tests.
• Release first version of new results server which will be the home for Mozmill functional test reporting and future home of Litmus test result reporting and Fennec automation test result reporting.
  • Complete Litmus user flow document for the test management side of Litmus.