SoftwareTesting

Overview

• Help Wanted

Dave Liebreich is interested in creating running code to test the mozilla codebase. Not just end-to-end automated tests, but unit tests, module tests, test harnesses, test reporting infrastructure, and anything else that lowers the barrier to creating new code that gives someone feedback.

Dave needs help. The codebase is huge. The community is huge. The task is huge.

This page is the first cut at collecting Dave's plans and investigations.

Guiding principles

• Testing is good
• Testing provides technical information about the product/system under test
• There are lots of different types of testing, each providing a different type of information and involving different trade-offs
• replacing a test that is useful to person A with a test that is useful to person B but not useful to person A is bad - just add the new test
• automated developer testing works only if the developers are writing the tests.
  • if developer finds something that is difficult to test, then developer can make code more testable
  • if code is more testable, then tests can be more powerful

Projects

• Catalog all existing test suites and/or test cases
  • special page for security tests
• investigate existing testing tools
  • watir (possibly using jssh)
  • xulunit, jsunit, jsunit (yes, there are 2), xunit, fit
• create test infrastructure architecture
  • common invocation and reporting formats for all tests
    • or choose 1 of no more than 3, if 1-size-does-not-fit-all
  • put docs on wiki, code in cvs
  • follow list of principles for automated testing
    • data-driven (data separate from test code)
    • Keyword driven
      • action keywords
      • fixtures to implement action keywords
    • focus on oracle (how do we know the test passed or failed)
    • do not rule out computer-assisted manual testing (no automated oracle)
• Community outreach
  • identify people in the mozilla community who are already test-infected
  • identify test-infected people outside the mozilla community and ask them to help
  • hit up grad students who can make this work part of their academic projects