Testrunner

From MozillaWiki
Jump to navigation Jump to search

Introduction

Testrunner is a test development tool built on top of Bugzilla, our installation of which is located at http://testrunner.mozilla.org (t.m.o.) --where QA (quality assurance) can create, modify and monitor test plans, test cases, test runs and test results.

Examples of other Testrunner installations:

At present the tests in Testrunner are manual, with a black-box persepective.

To-do list

  1. Upgrade from Testrunner 0.2 to 0.6 (or whatever the most recent version).
  2. Migrate existing test plans, functional groups and test cases into the more recent Testrunner version. Also, ensure that they remain accessible and useable in the new version.
    • Note that the old, pre-Bugzilla Component Makeover components had been used. We'll need to use the latest component organization.
  3. More seamless integration with the https://bugzilla.mozilla.org (b.m.o.) database, such as
    1. Allowing bug links in test cases and results (regardless of result status).
    2. Cross-referencing products and components from b.m.o. into Testrunner, instead of recreating them.
  4. Define (and publically document) usage guidelines for Testrunner, such as describing priviledges, outlining typical workflow, etc. This might also include a tutorial (as time permits).
  5. Improve access control:
    • Define who can create testplans, which could differ from those who can modify test plans.
    • Same goes for test cases, functional groups and components.
    • For test runs, control who can modify which runs, but still have flexibility to allow >1 user per active run —particularly useful for long test runs like BFT's.
  6. Improve Testrunner UI:
    1. Clarify navigation amongst test plans, test cases, test runs and test results. (Might be resolved in a more recent Testrunner build.)
    2. Improve maintenance, creation/editing and monitoring of test plans, test cases, test runs and test results.
    3. Allow for easier entry of comments for test runs and test cases.
  7. Integrate with Mister, the proposed tool to integrate https://update.mozilla.org (u.m.o.), Bouncer, the build systems (including Tinderbox/Bonsai), QA (noteably Testrunner and b.m.o.) and the release process.
  8. Integrate with other QA tools (Eggplant? Smoketest extension?)
  9. Integrate with Talkback (tracking build ids, test results, crash bugs, etc.)

Known issues and bugs

Some of these are observed in t.m.o., in which case they might've been resolved in a more recent Testrunner build.

  1. (t.m.o.) We found that the old style products and components are being used, and currently hinder organization of test cases such as those for Seamonkey. (More or less point (2) under "To-do list" above.)
  2. (t.m.o.) In spite of a case being in the "confirmed" state, a few such cases still don't appear in a test run. Why?
  3. (t.m.o.) Unable to delete cases (and perhaps functional groups, components, etc.) when there's a test run including that cases. Might need a more obvious-looking "disabled" state?