QA/TDAI/Goals/2009-Q2: Difference between revisions

← Older edit

QA/TDAI/Goals/2009-Q2 (view source)

Revision as of 15:08, 22 June 2009

1,983 bytes added , 22 June 2009

→‎Test Development

Ctalbert

Confirmed users

3,816

edits

@@ Line 1: / Line 1: @@
-{{draft}}
+This is both the agenda and the goals landing page for the Test Development and Automation team for Q2 2009.
-This is both the agenda and the goals landing page for the Test Development and Automation team for Q2 2009.  This is not official yet.  It is in draft form.
+= High Level Goals =
+* Increase visibility and expand automation by completing the build integration work for JS Ref Test, Mozmill, and Maemo automation harnesses.
+** ''JS Ref Test is at risk, mozmill is ok, maemo is ok.''
+* {{done|}}Support the 1.9.2 feature and performance work with functional test writing and aid in detailing/specifying new performance metrics (work with Fx Front End and Platform).
+** {{done|}} ''We completed work with networking performance, it is unclear if help is needed for front end, and mobile perf work is ongoing.''
+* Expand our test coverage by recruiting developers and community to write targeted tests in critical areas and by devoting one day every other week to publicly work on test development for all available harnesses.
+** {{done|}} ''Have not done as good a job at this as we would have liked early in the quarter due to 3.5 slippage, but will make up for it toward the end.''
+* Maintain and improve test tools to better enable people to do both manual and automated testing.
+** QAC, Mozmill, and the Results server are the tools we aim to update this quarter.
+** {{ok|}} Mozmill is ok.  {{risk|}} QAC is at risk.
 = Test Automation =
-'''Proposal (ctalbert)''': Greening Tinderboxes
+'''Focus:''' Improve the test automation system at harness level.
-* Institute a measuring system for tinderbox realibility
-* Debug the known, observed, random failures, create a plan for handling random failures going forward.
-'''Proposal (ctalbert)''': Close loop on Automation Suites
+'''Goal:''' Increase visibility and expand automation by completing the build integration work for JS Ref Test, Mozmill, and Maemo automation harnesses.
-* JS Ref test framework landing in Q2 {{bug|469718}}
-* Enable Build/Test infrastructure to run Mozmill tests (unit tests at least)
-** Have Mozmill tests running automatically, reporting to results server and/or staging tboxes (as appropriate)
-* Integrate Maemkit with Tinderbox, hand off to build
-'''Proposal (jmaher)''': Research and develop tools to run unittests on Windows Mobile.  This includes remote launching and refactoring existing tools
+''' Tactics:'''
-* Prototype in Q2
+* {{done|}}Institute a measuring system for tinderbox realibility
-* formally finished in Q3
+* {{done|}}Debug the known, observed, random failures, create a plan for handling random failures going forward.
+* {{risk|}}Land JS Ref test framework {{bug|469718}} - ''at risk due to 3.5 slip, getting build resources is hard and this feature is coming in late''
+* {{ok|}}Enable Build/Test infrastructure to run Mozmill tests (unit tests at least)
+** {{miss|}}Have Mozmill tests running automatically, reporting to results server and/or staging tboxes (as appropriate)
+* {{done|}}Integrate Maemkit with Tinderbox, hand off to build
+'''Other Harness Level Projects'''
+* {{done|}}Research and develop tools to run unittests on Windows Mobile.  This includes remote launching and refactoring existing tools
+** Prototype in Q2
+** formally finished in Q3
+* {{ok|}}Create a method for writing Fennec specific automated mochitests, xpcshell tests, reftests etc.
 = Test Development =
-'''Proposal (ctalbert)''': Expanding Test Coverage
+'''Focus:'''Expand test coverage across all harnesses and in all areas where more test coverage is needed.
-* Shared Team Goal to increase test coverage by X amount, where X is a measurement of reducing in-test-suite-? areas and improving code coverage numbers on previously untouched pieces of code.
-** Use code coverage results to identify functions within each team member's code area that are not covered well (< 40%)
+'''Goals:'''
-** Attempt to identify any in-test-suite-? bugs that also address this area
+* ''Goal 1:'' Support the 1.9.2 feature and performance work with functional test writing and aid in detailing/specifying new performance metrics (work with Fx Front End and Platform).
-** Create a list of the Y most wanted tests from the intersection of these points
+* ''Goal 2:'' Expand our test coverage by recruiting developers and community to write targeted tests in critical areas and by devoting one day every other week to publicly work on test development for all available harnesses.
-** Write those tests to target those lines of code and/or the address the concerns raised w.r.t. to the in-test-suite-? flag
-*** '''NOTES:'''
-*** maybe make this based on time as we don't know what the possible numbers will be
-*** seems like a good approach, not certain how much time it will take to come up with the list
-'''Proposal (jmaher)''': Define and implement method for Fennec specific tests (mochitest, chrome, browser-chrome, reftest, crashtest, xpcshell).
+'''Tactics:'''
-** harness related really, to fabricate mobile specific tests
-'''Proposal (jmaher)''': Develop tests which simulate Fennec specific hardware (such as hardkey press) to increase automated coverage and reduce litmus requirements
+Goal 1:
-** need this as part of the above goal (start this in Q2, probably more of an ongoing item)
+* {{done|}}Be involved in planning discussions for our areas of focus
+* {{ok|}}Draft test priority documents detailing the new features we will concentrate on for functional testing, get at developer review of those
+* {{miss|}}Write those functional tests
+** - ''3.5 slip has prevented us from concentrating as much here as we had anticipated''
+* {{done|}}Aid Mobile team as needed with their performance testing (seems like Taras is doing much of the heavy lifting?)
+* {{drop|}}Gather requirements and create mechanism for Firefox front end team's user performance metrics from "dirty" profiles - ''not sure how much this is still needed, and marking dropped as this item has been pushed into q3 by 3.5''
+* {{done|}}Gather requirements and gather platform performance metrics surrounding network performance for 3.5
-* do the new invalidation reftests that roc asked us for.
+Goal 2:
+* {{done|}}Create a list of 'hot-spots' where we need more testing and devote one day every other week to writing test cases each in our specific areas.  Determine the 'hot-spots' by analyzing:
+*** {{done|}}Query recent in-test-suite-? flags to find areas where people have flagged areas for more test development
+*** {{done|}} Use those areas to drill down into relevant files and determine where to concentrate test cases based on areas where we have low/incomplete code coverage
+*** {{done|}}Prioritize recently fixed security bugs for inclusion on the hot-spot list
+** {{done|}}Create the 'hot-spot' list for each area - JS, Content, Layout/Gfx, Mobile/Ffx Frontend
+** {{drop|}}Bring the list to developers to get them to help create tests - ''With the unrelenting focus on code blockers for 3.5 we did not feel it appropriate to bug developers with this''
-* Put the security tests into part of the "expanding test coverage" goal up above, these will be highest priority (a subset of the total set of security bugs) for those lists before other backlog items
+** Not having done this before, use a time based approach.  Next time we will look at number of test cases written this quarter and see how to change this goal going forward
-** writing tests for security backlog bugs how to address...?
-** codifying the security bugs into crashtests/etc test harnesses.
 = Community Leadership =
-'''Proposal (ctalbert)''': Use the "Most Wanted List" to expand code coverage
+'''Focus:'''Broaden the Test Development Team's reach and impact by recruiting more people to help and being more public in all we do.
-* Take the "Expanding Test Coverage" goal's "most wanted list" in each area and invite QA Community members (via QMO) and Development Community Members to help address these areas.  This is crucial as most of the time we'll need the dev community to tell us how to write these tests anyway, so it makes sense for them to be working on these alongside us.
-'''Proposal (jmaher)''': Develop a well defined set of tools/wishlist (thinking litmus, qac, mozmill features) that we can have available for summer internships, summer coding challenges or school projects.  Something that can increase our set of tools with features that help Mozilla as well as the general community if they wish to reuse our tools.
+'''Goal:'''Hold test development days where we publicly focus on writing tests on our 'hot-list' for each of our different test harnesses.  Invite community and developers to join us.
-* tools that focus on a area of testing that are harnessing web developers.  Enabling people to help people work with us, js framework unit tests (mikeal)
+'''Tactics:'''
+* {{ok|}}Use our 'hot-spot' list mentioned above as a resource of areas to work from when developing new tests
+* {{ok|}}Other avenues might be established by creating a well defined set of tools/wishlist projects (including detailed, concrete steps we need people to take) that we want help with: Litmus Redesign, Qac, Mozmill Helper Functions, Mozmill Features, Community Reftesting, etc.
 = Test Tools =
-'''Proposal (ctalbert)''': Release a redesigned QAC
+'''Focus:''' Create, maintain, and improve testing tools needed for ourselves and the larger testing community
-'''Proposal (ctalbert)''': Mozmill 1.2 (or 1.1.1) Maintenance Release
-* bug fixes, no major feature work
-** how can we ensure that the QAE team hits their automation goals?
-*** ctalbert takes this up with tony.
-*** create examples of tests?
-'''Proposal (ctalbert)''': Results Server Should be Reporting Mozmill Results
-* takes results, not up.  Need to get it running in Q2
-* needs view, might need adam's time for UI work
-* should be able to get results from runs that are both on tinderbox and not on tinderbox
-'''Proposal (jmaher)''': Results Server Should be Reporting Fennec unittest Results.  This is the ability to store results for various test runs and a a tool can query the result server to display the differences in tests run on fennec vs firefox.
+'''Goal:'''Maintain and improve test tools to better enable people to do both manual and automated testing
-* need JSON to put into the database
-* Need to work out indexes
-* Need to create a view for the results
-* Q2 :D
-= Personal Goals =
+'''Tactics:'''
-== Clint ==
+* {{risk|}}Provide a new QAC to the testing community. Focus on fixing usability issues with the tool so that people can more easily use that tool in their testing of Firefox.
-* Help achieve the "Greener Tinderbox" goal by leading that project
+* {{ok|}}Code and release Mozmill 1.2, addressing issues that block some segments of testing such as A11y and restartability
-* Do 10 phone screens to find candidates to fill our top two Test Dev Positions
+** {{ok|}}Support the QAE team by writing Mozmill Helper Functions as needed to make test writing easier.(Full set of bugs on this unknown, expect to write between 3 and 7 helper functions over the quarter, some may actually be folded into the main product if appropriate).
-* Aid with the "Expanding Test Coverage" by working in the Content area until we get help there
+** {{done|}}Tailor the JS Learning Project for Mozmill test writing in order to better enable QAE to write Mozmill tests.
-* Continue to lower the barrier to entry and create visibility for Test Dev by blogging biweekly and working with QMO
+* {{risk|}}Release first version of new results server which will be the home for Mozmill functional test reporting and future home of Litmus test result reporting and Fennec automation test result reporting. - ''This is on hold waiting on IT resources, blocked by the 650 castro move''
+** {{ok|}}Complete Litmus user flow document for the test management side of Litmus.