|
|
| Line 1: |
Line 1: |
| = Overview =
| | In [http://www.kaner.com/pdfs/TheOngoingRevolution.pdf The Ongoing Revolution in Software Testing (pdf)], [http://www.kaner.com Cem Kaner] defines testing as |
| * Help Wanted
| |
|
| |
|
| [[User:Davel|Dave Liebreich]] is interested in creating running code to test the mozilla codebase. Not just end-to-end automated tests, but unit tests, module tests, test harnesses, test reporting infrastructure, and anything else that lowers the barrier to creating new code that gives someone feedback. | | <blockquote cite="http://www.kaner.com/pdfs/TheOngoingRevolution.pdf">[...] a technical investigation of a product, done to expose quality-related information.</blockquote> |
|
| |
|
| Dave needs help. The codebase is huge. The community is huge. The task is huge.
| | This page is a starting point to find, learn about, and track technical investigations of the Mozilla codebase. |
|
| |
|
| This page is the first cut at collecting Dave's plans and investigations.
| | Please note: this catalog is not limited to investigations done by members of the Mozilla qa team and community. |
|
| |
|
| = Guiding principles =
| | * [[/Scratchpad]] |
| * Testing is good | |
| * Testing provides technical information about the product/system under test
| |
| * There are lots of different types of testing, each providing a different type of information and involving different trade-offs
| |
| * replacing a test that is useful to person A with a test that is useful to person B but not useful to person A is bad - just add the new test
| |
| * automated developer testing works only if the developers are writing the tests.
| |
| ** if developer finds something that is difficult to test, then developer can make code more testable
| |
| ** if code is more testable, then tests can be more powerful
| |
| * feedback from user community is unstructured and somewhat unpredictable, but incredibly useful. We should not extinguish this valuable channel of information.
| |
| * developer unit tests are typically not-very-powerful tests. The value comes from
| |
| ** immediate and precise feedback - written close to the code
| |
| ** there are typically lots of them
| |
| ** they function as executable specifications for the code, and can replace some documentation and usage examples
| |
| * we need involvement from toolsmiths and developers and testers
| |
| ** toolsmiths maintain the infrastructure, and find/adapt/use tools to solve problems
| |
| ** developers write developer tests
| |
| ** testers write end-user tests
| |
| ** each group shares domain knowledge with the others
| |
| | |
| = Projects =
| |
| * Catalog all existing test suites and/or test cases
| |
| ** special page for security tests
| |
| * investigate existing testing tools
| |
| ** Watir (possibly using JSSh)
| |
| ** xulUnit, jsUnit, jsUnit (yes, there are 2), xUnit, FIT
| |
| * create test infrastructure architecture
| |
| ** common invocation and reporting formats for all tests
| |
| *** or choose 1 of no more than 3, if 1-size-does-not-fit-all
| |
| ** put docs on wiki, code in cvs
| |
| ** follow list of principles for automated testing
| |
| *** data-driven (data separate from test code)
| |
| *** Keyword driven
| |
| **** action keywords
| |
| **** fixtures to implement action keywords
| |
| *** focus on oracle (how do we know the test passed or failed)
| |
| *** do not rule out computer-assisted manual testing (no automated oracle)
| |
| *** use model-based test scripting when possible
| |
| **** allows for high volume automated testing
| |
| **** can walk the state transitions and detect failures other than asserts or crashes
| |
| * Community outreach
| |
| ** identify people in the mozilla community who are already test-infected
| |
| ** identify test-infected people outside the mozilla community and ask them to help
| |
| ** hit up grad students who can make this work part of their academic projects
| |