We all want quality to happen. How can we help? What is being run? What can I run something with? What does any of this mean? Not all questions are answered, but some.
This document has three sections:
- Test Harnesses - tools, designed to run tests. A harness may be targeted to a specific set of tests, or general to many kinds of tests. What test do you want to see run? Different harnesses could be used to run a given test. Which harness is better for which kinds of testing? kinds of testing?
- Test Suites - arbitrary groupings of test cases, roughly organized by functional area
- Test Infrastructure - the environments the test tools and test cases run in. This includes machine, operating system and physical location (ex: QA farm vs QA lab vs someone's desktop)
Test Harnesses
| Test Harness Name | Brief Description | Language Used | Status of Harness | Notes |
| MochiKit | ajaxified, high cool-ness factor, currently being made to work in automated test harnesses. | JavaScript | Invocation by buildbot on QA machines is being set up. Invocation of tests by inidividuals on built trunk is possible with setup. | [MochiTest on devmo] |
| jsunit | may be less cool, may not be made to work in automation, but a "unit" and proud of it! | JavaScript | ||
| litmus | manual test cases. How do you do that voodoo that you do so well? | English, German, French, etc. | [main site] | |
| xpcshell | lots of under-the-hood XPCOM goodness! | some JavaScript, much C++ | [xpcshell on devmo] | |
| SpiderMonkey | bc likes it. It must be righteous. | JavaScript | [bc's page] | |
| reftest | New kid on the block, visual-ish comparisons via JavaScript and Canvas | JavaScript | runnable on built trunk | [reftest on devmo] |
| jssh | a javascript shell. What more can you say? | JavaScript | described as "largely defunct" | |
| 'Browser and Eyeballs' | The most common test harness in use today. | wetware | There is a new tester born every day! :-) | |
| Download Checker | listed as "run manually by QA" ??? | [wiki page] | ||
| Software Update Checker | listed as "run manually by QA" ??? | [wiki page] | ||
| l10n Search Verifier | listed as "run manually by QA" ??? | [wiki page] |
Test Suites
| Test Suite Name | Brief Description | Test Framework/Infrstructure Used | Status | Notes | Owner |
| JS tests | Custom framework. Runs on QA Farm. | Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007 | |||
| Security tests | Custom framework. Runs on QA Farm. | Semi-automated. Plans to fully automate with Eggplant in Q1 2007 | |||
| Automated Smoke Tests - FFx 1.5 | Eggplant. Runs on Tracy's machines. | Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move executiion to QA Farm in Q1 2007 | |||
| Automated Smoke Tests - FFx 2.0 | Eggplant. Runs on Tracy's machines. | Fully automated. Completed porting 1.5 tests to the 3 platforms for 2.0 in Q4 2006. Plan to move executiion to QA Farm in Q1 2007 | |||
| Basic Functionality Tests (BFT) | in Litmus | ||||
| Full Functionality Tests (FFT) | in Litmus | ||||
| Localization Tests | in Litmus | ||||
| Software Update Suite | in Litmus | ||||
| Browser/Content | custom JavaScript driver? | browser/components/places=9 browser/components/safebrowsing=1 browser/components/sessionstore=1 content/base=9 content/events=0 content/html/content=15 content=10 content/xml=35 content/xslt=23 content/xslt/XSLTMark=5 |
|||
| Buster | re-purposing of the Adobe Xalan tests. Driver is in mozilla, but test content gets re-downloaded. | custom driver in repository | content/xslt/buster | ||
| Database | will probably be made obsolete by db/sqlite3 | custom executable | dbm=2 | ||
| DOM | browser and eyeballs? | dom/tests/html=1 dom/tests/js=19 |
|||
| Embedding | ??? | embedding/browser/activex=87 embedding/browser/gtk=5 embedding/browser/photon=1 embedding/browser/qt=7 embedding/qa/jstests=88 embedding/qa/jstests/accessibility=53 embedding/qaembed=55 embedding/qaembed=55 embedding/apitest=2 embedding=113 |
|||
| Extensions | ??? | extensions/canvas3d=6 extensions/java/xpcom=10 extensions/metrics=5 extensions/mono=5 extensions/python=31 extensions/schema-validation=9 extensions/spatialnavigation=5 extensions/universalchardet=2 extensions/webdav=1 |
|||
| Graphics | custom executable/C++ | gfx=8 | |||
| I18N | custom executable/C++ | intl=37 | |||
| XPConnect | custom executable/C++ | js/src/xpconnect=142 | |||
| Layout/HTML | "the old regression test suite" and reftest | layout/html=2461 layout/reftests=360 layout/svg=2 layout/tools/reftest=6 layout/tools=4 |
|||
| Math/ML | "browsers and eyeballs" | also using w3c.org site content | layout/mathml=7 | ||
| ??? | modules/libjar=4 modules/oji=758 modules/plugin=63 |
||||
| Networking | custom executable/C++ | netwerk=121 | |||
| NSPR | custom executable/C | nsprpub/lib=3 | |||
| HTML Parsing | custom "TestParse" executable | parser/htmlparser=327 | |||
| RDF | custom executable/C++ | rdf=11 | |||
| Security/NSS | custom executable and "make check" | security/dbm=0 security/nss=56 |
|||
| XPCOM | custom executable/C++ | xpcom/proxy=6 xpcom=66 |
|||
| XPFE | JavaScript/XULRunner? | xpfe=3 | |||
| XPInstall | ??? | xpinstall=3 |
Test Infrastructures
| Test Infrastructure Name | Brief Description | Harnesses and Test Suites in the Infrastructure | Status of the Instructure | Notes |
| tinderboxen | Continuous build engine for Mozilla source, much chronicled in song and fable. | "make check", the results of which may not be verified | see dynamic Tinderboxen display | |
| buildbots | Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins. Running make check, reftest and mochitest | |||
| QA Farm | some number of Windows, Linux and Macs. Details are not important. They should be a black box resource for developers to say "test this". |