QA/TDAI/Goals/2009-Q2: Difference between revisions

From MozillaWiki
< QA‎ | TDAI
Jump to navigation Jump to search
No edit summary
Line 4: Line 4:


= Test Automation =
= Test Automation =
'''Proposal (ctalbert)''': Greening Tinderboxes
'''Focus:''' Improve the test automation system at harness level.
'''Goal:''' Increase visibility and expand automation by completing the circle on JS testing, Mozmill, and Maemo.
''' Tactics:'''
* Institute a measuring system for tinderbox realibility
* Institute a measuring system for tinderbox realibility
* Debug the known, observed, random failures, create a plan for handling random failures going forward.
* Debug the known, observed, random failures, create a plan for handling random failures going forward.
'''Proposal (ctalbert)''': Close loop on Automation Suites
* JS Ref test framework landing in Q2 {{bug|469718}}
* JS Ref test framework landing in Q2 {{bug|469718}}
* Enable Build/Test infrastructure to run Mozmill tests (unit tests at least)
* Enable Build/Test infrastructure to run Mozmill tests (unit tests at least)
Line 14: Line 14:
* Integrate Maemkit with Tinderbox, hand off to build
* Integrate Maemkit with Tinderbox, hand off to build


'''Proposal (jmaher)''': Research and develop tools to run unittests on Windows Mobile.  This includes remote launching and refactoring existing tools
'''Other Harness Level Projects'''
* Prototype in Q2
* Research and develop tools to run unittests on Windows Mobile.  This includes remote launching and refactoring existing tools
* formally finished in Q3
** Prototype in Q2
** formally finished in Q3
* Create a method for writing Fennec specific automated mochitests, xpcshell tests, reftests etc.


= Test Development =
= Test Development =
'''Proposal (ctalbert)''': Expanding Test Coverage
'''Focus:'''Expand test coverage across all harnesses and in all areas where more test coverage is needed.
* Shared Team Goal to increase test coverage by X amount, where X is a measurement of reducing in-test-suite-? areas and improving code coverage numbers on previously untouched pieces of code.
'''Goal:'''
** Use code coverage results to identify functions within each team member's code area that are not covered well (< 40%)
* Expand our test coverage by recruiting developers to write targeted tests in critical areas and by devoting 10 hours a week of our time to do the same for the backlog of bugs and areas that most need tests.
** Attempt to identify any in-test-suite-? bugs that also address this area
* Research and create a set of testing priorities for 1.9.2 features on each of JS, Layout/GFX, Content, Mobile areas.
** Create a list of the Y most wanted tests from the intersection of these points
 
** Write those tests to target those lines of code and/or the address the concerns raised w.r.t. to the in-test-suite-? flag
'''Tactics:'''
*** '''NOTES:'''
* Goal 1:
*** maybe make this based on time as we don't know what the possible numbers will be
** Create a list of 'hot-spots' where we need more testing and devote 10 hours a week to writing test cases each in our specific areas.  Determine the 'hot-spots' by analyzing:
*** seems like a good approach, not certain how much time it will take to come up with the list
**** Query recent in-test-suite-? flags to find areas where people have flagged areas for more test development
 
**** Use those areas to drill down into relevant files and determine where to concentrate test cases based on areas where we have low/incomplete code coverage
'''Proposal (jmaher)''': Define and implement method for Fennec specific tests (mochitest, chrome, browser-chrome, reftest, crashtest, xpcshell). 
**** Prioritize recently fixed security bugs for inclusion on the hot-spot list
** harness related really, to fabricate mobile specific tests
*** Create the 'hot-spot' list for each area - JS, Content, Layout/Gfx, Mobile
 
*** Bring the list to developers to get them to help create tests
'''Proposal (jmaher)''': Develop tests which simulate Fennec specific hardware (such as hardkey press) to increase automated coverage and reduce litmus requirements
*** Not having done this before, use a time based approach. Next time we will look at number of test cases written this quarter and see how to change this goal going forward
** need this as part of the above goal (start this in Q2, probably more of an ongoing item)
* Goal 2:
 
** Be involved in planning discussions for our areas of focus
* do the new invalidation reftests that roc asked us for.
** Draft test priority documents, get developer review of those
 
* Put the security tests into part of the "expanding test coverage" goal up above, these will be highest priority (a subset of the total set of security bugs) for those lists before other backlog items
** writing tests for security backlog bugs how to address...?
** codifying the security bugs into crashtests/etc test harnesses.


= Community Leadership =
= Community Leadership =
'''Proposal (ctalbert)''': Use the "Most Wanted List" to expand code coverage
'''Focus:'''Broaden the Test Development Team's reach and impact by recruiting more people to help and being more public in all we do.
* Take the "Expanding Test Coverage" goal's "most wanted list" in each area and invite QA Community members (via QMO) and Development Community Members to help address these areas. This is crucial as most of the time we'll need the dev community to tell us how to write these tests anyway, so it makes sense for them to be working on these alongside us.
'''Goal:'''Create 4 avenues for people to quickly get involved and begin contributing to test development work.
 
'''Tactics:'''
'''Proposal (jmaher)''': Develop a well defined set of tools/wishlist (thinking litmus, qac, mozmill features) that we can have available for summer internships, summer coding challenges or school projects.  Something that can increase our set of tools with features that help Mozilla as well as the general community if they wish to reuse our tools.
* One avenue is the 'hot-spot' list mentioned above
 
* Other avenues might established by creating a well defined set of tools/wishlist projects that we want help with: Litmus Redesign, Qac, Mozmill Helper Functions, Mozmill Features, Community Reftesting, etc.
* tools that focus on a area of testing that are harnessing web developers.  Enabling people to help people work with us, js framework unit tests (mikeal)


= Test Tools =
= Test Tools =
Line 75: Line 72:
* Aid with the "Expanding Test Coverage" by working in the Content area until we get help there
* Aid with the "Expanding Test Coverage" by working in the Content area until we get help there
* Continue to lower the barrier to entry and create visibility for Test Dev by blogging biweekly and working with QMO
* Continue to lower the barrier to entry and create visibility for Test Dev by blogging biweekly and working with QMO
== Jmaher ==
* Part of Test Development goals:
** Develop tests which simulate Fennec specific hardware (such as hardkey press) to increase automated coverage and reduce litmus requirements
== mw22 and clint ==
* Part of Test Development Goals:
** Do the invalidation reftests that Roc asked us for

Revision as of 19:26, 27 March 2009

Draft-template-image.png THIS PAGE IS A WORKING DRAFT Pencil-emoji U270F-gray.png
The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly.
If you have any questions or ideas, please add them as a new topic on the discussion page.

This is both the agenda and the goals landing page for the Test Development and Automation team for Q2 2009. This is not official yet. It is in draft form.

Test Automation

Focus: Improve the test automation system at harness level. Goal: Increase visibility and expand automation by completing the circle on JS testing, Mozmill, and Maemo. Tactics:

  • Institute a measuring system for tinderbox realibility
  • Debug the known, observed, random failures, create a plan for handling random failures going forward.
  • JS Ref test framework landing in Q2 bug 469718
  • Enable Build/Test infrastructure to run Mozmill tests (unit tests at least)
    • Have Mozmill tests running automatically, reporting to results server and/or staging tboxes (as appropriate)
  • Integrate Maemkit with Tinderbox, hand off to build

Other Harness Level Projects

  • Research and develop tools to run unittests on Windows Mobile. This includes remote launching and refactoring existing tools
    • Prototype in Q2
    • formally finished in Q3
  • Create a method for writing Fennec specific automated mochitests, xpcshell tests, reftests etc.

Test Development

Focus:Expand test coverage across all harnesses and in all areas where more test coverage is needed. Goal:

  • Expand our test coverage by recruiting developers to write targeted tests in critical areas and by devoting 10 hours a week of our time to do the same for the backlog of bugs and areas that most need tests.
  • Research and create a set of testing priorities for 1.9.2 features on each of JS, Layout/GFX, Content, Mobile areas.

Tactics:

  • Goal 1:
    • Create a list of 'hot-spots' where we need more testing and devote 10 hours a week to writing test cases each in our specific areas. Determine the 'hot-spots' by analyzing:
        • Query recent in-test-suite-? flags to find areas where people have flagged areas for more test development
        • Use those areas to drill down into relevant files and determine where to concentrate test cases based on areas where we have low/incomplete code coverage
        • Prioritize recently fixed security bugs for inclusion on the hot-spot list
      • Create the 'hot-spot' list for each area - JS, Content, Layout/Gfx, Mobile
      • Bring the list to developers to get them to help create tests
      • Not having done this before, use a time based approach. Next time we will look at number of test cases written this quarter and see how to change this goal going forward
  • Goal 2:
    • Be involved in planning discussions for our areas of focus
    • Draft test priority documents, get developer review of those

Community Leadership

Focus:Broaden the Test Development Team's reach and impact by recruiting more people to help and being more public in all we do. Goal:Create 4 avenues for people to quickly get involved and begin contributing to test development work. Tactics:

  • One avenue is the 'hot-spot' list mentioned above
  • Other avenues might established by creating a well defined set of tools/wishlist projects that we want help with: Litmus Redesign, Qac, Mozmill Helper Functions, Mozmill Features, Community Reftesting, etc.

Test Tools

Proposal (ctalbert): Release a redesigned QAC

Proposal (ctalbert): Mozmill 1.2 (or 1.1.1) Maintenance Release

  • bug fixes, no major feature work
    • how can we ensure that the QAE team hits their automation goals?
      • ctalbert takes this up with tony.
      • create examples of tests?

Proposal (ctalbert): Results Server Should be Reporting Mozmill Results

  • takes results, not up. Need to get it running in Q2
  • needs view, might need adam's time for UI work
  • should be able to get results from runs that are both on tinderbox and not on tinderbox

Proposal (jmaher): Results Server Should be Reporting Fennec unittest Results. This is the ability to store results for various test runs and a a tool can query the result server to display the differences in tests run on fennec vs firefox.

  • need JSON to put into the database
  • Need to work out indexes
  • Need to create a view for the results
  • Q2 :D

Personal Goals

Clint

  • Help achieve the "Greener Tinderbox" goal by leading that project
  • Do 10 phone screens to find candidates to fill our top two Test Dev Positions
  • Aid with the "Expanding Test Coverage" by working in the Content area until we get help there
  • Continue to lower the barrier to entry and create visibility for Test Dev by blogging biweekly and working with QMO

Jmaher

  • Part of Test Development goals:
    • Develop tests which simulate Fennec specific hardware (such as hardkey press) to increase automated coverage and reduce litmus requirements

mw22 and clint

  • Part of Test Development Goals:
    • Do the invalidation reftests that Roc asked us for
Retrieved from "https://wiki.allizom.org/index.php?title=QA/TDAI/Goals/2009-Q2&oldid=136700"