QA/TDAI/Goals/2009-Q2

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

This is both the agenda and the goals landing page for the Test Development and Automation team for Q2 2009. This is not official yet. It is in draft form.

Test Automation

Proposal (ctalbert): Greening Tinderboxes

• Institute a measuring system for tinderbox realibility
• Debug the known, observed, random failures, create a plan for handling random failures going forward.

Proposal (ctalbert): Close loop on Automation Suites

• JS Ref test framework landing in Q2 bug 469718
• Enable Build/Test infrastructure to run Mozmill tests (unit tests at least)
  • Have Mozmill tests running automatically, reporting to results server and/or staging tboxes (as appropriate)
• Integrate Maemkit with Tinderbox, hand off to build

Proposal (jmaher): Research and develop tools to run unittests on Windows Mobile. This includes remote launching and refactoring existing tools

• Prototype in Q2
• formally finished in Q3

Test Development

Proposal (ctalbert): Expanding Test Coverage

• Shared Team Goal to increase test coverage by X amount, where X is a measurement of reducing in-test-suite-? areas and improving code coverage numbers on previously untouched pieces of code.
  • Use code coverage results to identify functions within each team member's code area that are not covered well (< 40%)
  • Attempt to identify any in-test-suite-? bugs that also address this area
  • Create a list of the Y most wanted tests from the intersection of these points
  • Write those tests to target those lines of code and/or the address the concerns raised w.r.t. to the in-test-suite-? flag
    • NOTES:
    • maybe make this based on time as we don't know what the possible numbers will be
    • seems like a good approach, not certain how much time it will take to come up with the list

Proposal (jmaher): Define and implement method for Fennec specific tests (mochitest, chrome, browser-chrome, reftest, crashtest, xpcshell).

• • harness related really, to fabricate mobile specific tests

Proposal (jmaher): Develop tests which simulate Fennec specific hardware (such as hardkey press) to increase automated coverage and reduce litmus requirements

• • need this as part of the above goal (start this in Q2, probably more of an ongoing item)

• do the new invalidation reftests that roc asked us for.

• Put the security tests into part of the "expanding test coverage" goal up above, these will be highest priority for those lists before other backlog items
  • writing tests for security backlog bugs how to address...?
  • codifying the security bugs into crashtests/etc test harnesses.

Community Leadership

Proposal (ctalbert): Use the "Most Wanted List" to expand code coverage

• Take the "Expanding Test Coverage" goal's "most wanted list" in each area and invite QA Community members (via QMO) and Development Community Members to help address these areas. This is crucial as most of the time we'll need the dev community to tell us how to write these tests anyway, so it makes sense for them to be working on these alongside us.

Proposal (jmaher): Develop a well defined set of tools/wishlist (thinking litmus, qac, mozmill features) that we can have available for summer internships, summer coding challenges or school projects. Something that can increase our set of tools with features that help Mozilla as well as the general community if they wish to reuse our tools.

• tools that focus on a area of testing that are harnessing web developers. Enabling people to help people work with us, js framework unit tests (mikeal)

Test Tools

Proposal (ctalbert): Release a redesigned QAC

Proposal (ctalbert): Mozmill 1.2 (or 1.1.1) Maintenance Release

• bug fixes, no major feature work
  • how can we ensure that the QAE team hits their automation goals?
    • ctalbert takes this up with tony.
    • create examples of tests?

Proposal (ctalbert): Results Server Should be Reporting Mozmill Results

• takes results, not up. Need to get it running in Q2
• needs view, might need adam's time for UI work
• should be able to get results from runs that are both on tinderbox and not on tinderbox

Proposal (jmaher): Results Server Should be Reporting Fennec unittest Results. This is the ability to store results for various test runs and a a tool can query the result server to display the differences in tests run on fennec vs firefox.

• need JSON to put into the database
• Need to work out indexes
• Need to create a view for the results
• Q2 :D

Personal Goals

Clint

• Help achieve the "Greener Tinderbox" goal by leading that project
• Do 10 phone screens to find candidates to fill our top two Test Dev Positions
• Aid with the "Expanding Test Coverage" by working in the Content area until we get help there
• Continue to lower the barrier to entry and create visibility for Test Dev by blogging biweekly and working with QMO