201
edits
SoftwareTesting:Scratchpad: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
= Overview = | |||
* Help Wanted | |||
[[User:Davel|Dave Liebreich]] is interested in creating running code to test the mozilla codebase. Not just end-to-end automated tests, but unit tests, module tests, test harnesses, test reporting infrastructure, and anything else that lowers the barrier to creating new code that gives someone feedback. | |||
Dave needs help. The codebase is huge. The community is huge. The task is huge. | |||
This page is the first cut at collecting Dave's plans and investigations. | |||
= Guiding principles = | |||
* Testing is good | |||
* Testing provides technical information about the product/system under test | |||
* There are lots of different types of testing, each providing a different type of information and involving different trade-offs | |||
* replacing a test that is useful to person A with a test that is useful to person B but not useful to person A is bad - just add the new test | |||
* automated developer testing works only if the developers are writing the tests. | |||
** if developer finds something that is difficult to test, then developer can make code more testable | |||
** if code is more testable, then tests can be more powerful | |||
* feedback from user community is unstructured and somewhat unpredictable, but incredibly useful. We should not extinguish this valuable channel of information. | |||
* developer unit tests are typically not-very-powerful tests. The value comes from | |||
** immediate and precise feedback - written close to the code | |||
** there are typically lots of them | |||
** they function as executable specifications for the code, and can replace some documentation and usage examples | |||
* we need involvement from toolsmiths and developers and testers | |||
** toolsmiths maintain the infrastructure, and find/adapt/use tools to solve problems | |||
** developers write developer tests | |||
** testers write end-user tests | |||
** each group shares domain knowledge with the others | |||
= Projects = | |||
* Catalog all existing test suites and/or test cases | |||
** special page for security tests | |||
* investigate existing testing tools | |||
** Watir (possibly using JSSh) | |||
** xulUnit, jsUnit, jsUnit (yes, there are 2), xUnit, FIT | |||
* create test infrastructure architecture | |||
** common invocation and reporting formats for all tests | |||
*** or choose 1 of no more than 3, if 1-size-does-not-fit-all | |||
** put docs on wiki, code in cvs | |||
** follow list of principles for automated testing | |||
*** data-driven (data separate from test code) | |||
*** Keyword driven | |||
**** action keywords | |||
**** fixtures to implement action keywords | |||
*** focus on oracle (how do we know the test passed or failed) | |||
*** do not rule out computer-assisted manual testing (no automated oracle) | |||
*** use model-based test scripting when possible | |||
**** allows for high volume automated testing | |||
**** can walk the state transitions and detect failures other than asserts or crashes | |||
* Community outreach | |||
** identify people in the mozilla community who are already test-infected | |||
** identify test-infected people outside the mozilla community and ask them to help | |||
** hit up grad students who can make this work part of their academic projects | |||