Testrunner
Jump to navigation
Jump to search
Introduction
Testrunner is a test development tool built on top of Bugzilla, our installation of which is located at http://testrunner.mozilla.org (t.m.o.) --where QA (quality assurance) can create, modify and monitor test plans, test cases, test runs and test results.
Examples of other Testrunner installations:
- Eduardo Fuentetaja's server running v0.6.1-beta: http://testrunner.citat.se/
- Opi's server running a recent CVS pull (Warning: Under development! Might go poof at any time, but ideally this is what Seamonkey QA might use): http://www2.le-bit.de/bugzilla/index.cgi
At present the tests in Testrunner are manual, with a black-box persepective.
To-do list
- Upgrade from Testrunner 0.2 to 0.6 (or whatever the most recent version).
- Migrate existing test plans, functional groups and test cases into the more recent Testrunner version. Also, ensure that they remain accessible and useable in the new version.
- Note that the old, pre-Bugzilla Component Makeover components had been used. We'll need to use the latest component organization.
- More seamless integration with the https://bugzilla.mozilla.org (b.m.o.) database, such as
- Allowing bug links in test cases and results (regardless of result status).
- Cross-referencing products and components from b.m.o. into Testrunner, instead of recreating them.
- Define (and publically document) usage guidelines for Testrunner, such as describing priviledges, outlining typical workflow, etc. This might also include a tutorial (as time permits).
- Improve access control:
- Define who can create testplans, which could differ from those who can modify test plans.
- Same goes for test cases, functional groups and components.
- For test runs, control who can modify which runs, but still have flexibility to allow >1 user per active run —particularly useful for long test runs like BFT's.
- Improve Testrunner UI:
- Clarify navigation amongst test plans, test cases, test runs and test results. (Might be resolved in a more recent Testrunner build.)
- Improve maintenance, creation/editing and monitoring of test plans, test cases, test runs and test results.
- Allow for easier entry of comments for test runs and test cases.
- Integrate with Mister, the proposed tool to integrate https://update.mozilla.org (u.m.o.), Bouncer, the build systems (including Tinderbox/Bonsai), QA (noteably Testrunner and b.m.o.) and the release process.
- Integrate with other QA tools (Eggplant? Smoketest extension?)
- Integrate with Talkback (tracking build ids, test results, crash bugs, etc.)
Known issues and bugs
Some of these are observed in t.m.o., in which case they might've been resolved in a more recent Testrunner build.
- (t.m.o.) We found that the old style products and components are being used, and currently hinder organization of test cases such as those for Seamonkey. (More or less point (2) under "To-do list" above.)
- (t.m.o.) In spite of a case being in the "confirmed" state, a few such cases still don't appear in a test run. Why?
- (t.m.o.) Unable to delete cases (and perhaps functional groups, components, etc.) when there's a test run including that cases. Might need a more obvious-looking "disabled" state?