Testrunner

Introduction

Testrunner is a test development tool built on top of Bugzilla, our installation of which is located at http://testrunner.mozilla.org (t.m.o.) --where QA (quality assurance) can create, modify and monitor test plans, test cases, test runs and test results.

Examples of other Testrunner installations:

• Eduardo Fuentetaja's server running v0.6.2-beta on 2.18: http://testrunner.citat.se/

• Opi's server running a recent CVS pull (0.6.2 05.09.05 on 2.19.2) (Warning: Under development! Might go poof at any time, but ideally this is what Seamonkey QA might use): http://www2.le-bit.de/bugzilla/index.cgi

At present the tests in Testrunner are manual, with a black-box persepective.

To-do list

1. Upgrade from Testrunner 0.2 to 0.6 (or whatever the most recent version). This includes bringing our local modifications to the new version.
   • In order to do this, it would be ideal to separate Testrunner and Bugzilla a little bit, to avoid administrative chaos and breakage when Bugzilla upgades. Step one with this would be to eliminate the patch that Testrunner applies to Bugzilla. We should make Testrunner use the template extension mechanism instead.
     • See Testrunner bug #1224354
     • Testrunner is badly behaved and copies its templates into template/en/default/tr_testrunner. This is evil.
     • We also need to figure out how to deal with the database. Testrunner is designed to share a database with Bugzilla, which seems just a bit cludgy and likely to cause trouble down the road.
2. Migrate existing test plans, functional groups and test cases into the more recent Testrunner version. Also, ensure that they remain accessible and useable in the new version.
   • Note that the old, pre-Bugzilla Component Makeover components had been used. We'll need to use the latest component organization.
3. More seamless integration with the https://bugzilla.mozilla.org (b.m.o.) database, such as
   • Allowing bug links in test cases and results (regardless of result status).
   • Cross-referencing products and components from b.m.o. into Testrunner, instead of recreating them.
4. Define (and publically document) usage guidelines for Testrunner, such as describing priviledges, outlining typical workflow, etc. This might also include a tutorial (as time permits).
   • Allow flags as an option, such a review request (for a test plan or case) or certification request (for a test run). Would be good to have some flexibility so that requests aren't required all the time, which might make the process slow or awkward. However, this would be useful for localization teams needing certification, or MoFo QA needing to certify release candidates at key points in the release cycle, etc.
5. Improve access control:
   • Right now (version 0.6), any user can create test runs and run tests, and we can give other users the following special permissions bits:
     • managetestplans - allows the user to create, edit, and delete test plans
     • edittestcases - allows the user to create, edit, and delete test cases
   • What this model really ought to be is permissions on a per-product/component basis, such that someone can create test plans and cases for SeaMonkey but not for Browser for instance.
   • Define who can create testplans, which could differ from those who can modify test plans.
   • Same goes for test cases, functional groups and components.
   • For test runs, control who can modify which runs, but still have flexibility to allow >1 user per active run —particularly useful for long test runs like BFT's.
6. Improve Testrunner UI:
   • Clarify navigation amongst test plans, test cases, test runs and test results. (Might be resolved in a more recent Testrunner build.)
   • Improve maintenance, creation/editing and monitoring of test plans, test cases, test runs and test results. It'd be nice to be able to change the order of test cases in a functional group, too.
   • Allow for easier entry of comments for test runs and test cases.
   • Display last-modified date and user who modified a test plan or test case.
7. Integrate with Mister, the proposed tool to integrate https://update.mozilla.org (u.m.o.), Bouncer, the build systems (including Tinderbox/Bonsai), QA (noteably Testrunner and b.m.o.) and the release process.
8. Integrate with other QA tools, notably more automated and/or white/grey-box oriented testing (Eggplant? Smoketest extension?)
9. Integrate with Talkback (tracking build ids, test results, crash bugs, etc.)
10. Purely cosmetic: Testrunner seems to dislike the idea of underlining its hyperlinks. This should be fixed.

Known issues and bugs

Some of these are observed in t.m.o., in which case they might've been resolved in a more recent Testrunner build.

1. (t.m.o.) We found that the old style products and components are being used, and currently hinder organization of test cases such as those for Seamonkey. (More or less point (2) under "To-do list" above.)
2. (t.m.o.) In spite of a case being in the "confirmed" state, a few such cases still don't appear in a test run. Why?
3. (t.m.o.) Unable to delete cases (and perhaps functional groups, components, etc.) when there's a test run including that cases. Might need a more obvious-looking "disabled" state?
4. (t.m.o.) Currently unable to change the order of test cases within a given functional group; frustrating when the order of tests matters.
5. (t.m.o.) Currently doesn't (easily) display the last-modified date or user who modified a given test plan, group or case.
6. (t.m.o.) If I accidentally mark a test case as failed, but meant to leave it as "not run," I cannot set it back to "not run."