|
THIS PAGE IS A WORKING DRAFT | 
|
| The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page. |
We all want quality to happen. How can we help? What is being run? What can I run something with? What does any of this mean? Not all questions are answered, but some.
This document has three sections:
- Test Harnesses - tools, designed to run tests. A harness may be targeted to a specific set of tests, or general to many kinds of tests. What test do you want to see run? Different harnesses could be used to run a given test. Which harness is better for which kinds of testing? kinds of testing?
- Test Suites - arbitrary groupings of test cases, roughly organized by functional area
- Test Infrastructure - the environments the test tools and test cases run in. This includes machine, operating system and physical location (ex: QA farm vs QA lab vs someone's desktop)
There are questions to be answered about any of our tests. If anyone has information or suggestions, please let us know.
- What tests do we have now, how current are they and how they should be run? Should they be expected to pass?
- What features or modules need tests to be written for them? What are the requirements of the testing that should happen?
- What tools do we have now that can be used to write tests? What tools outside of mozilla have people used to build tests and test tools.
- Of the tests currently in existence, which of them are outside the "mozilla.org" umbrella? Which use resources from outside? What is the reliability and accessibility of these resources? Are there issues which keep us from keeping local copies?
Any questions and suggestions are welcome!
Test Harnesses
| Test Harness Name | Brief Description | Language Used | Status of Harness | Notes |
| MochiTest | built by sayrer using the MochiKit framework | JavaScript | Automation being set up on QA Farm. Exec by community is possible with setup. | devmo |
| eggplant | uses VNC for device-level visual comparisons | Being used for automation of Smoke Tests suites in litmus | redstone | |
| litmus | Catalog of manual test cases. Being developed to allow community to run the manual smoke tests, BFT, FFT and L10n tests. GUI to capture results and provide feedback to the QA team. Reporting function to check test and platform coverage. | English, German, French, etc. | The place to be on Community Test Day! Still in refinement by zlipton and coop. 2445 test cases, 1994 tester accounts with 845 active | litmus |
| T* | perf tests, runs in tinderbox | running, passing, run by tinderbox | wiki | |
| xpcshell | used by 40+ tests, plans for use by many more | JavaScript | recent care and feeding, very usable | devmo |
| SpiderMonkey | crawls a site to specified depth, loading CSS, executing JS along the way. | JavaScript | used only for the js/tests by bc, et al, for JS testing | bc |
| reftest | comparison of two pages, via JavaScript and Canvas, for visual equivalency | JavaScript | runnable on built trunk | devmo |
| build/make | unit tests in source | "make check" is accepted standard, being integrated with buildbot | tinderbox |
Test Suites
| Test Suite Name | Brief Description | Test Harness(es) | Status | Notes | Owner |
| JS tests | well-developed suite of tests of the JavaScript language in js/tests | SpiderMonkey Runs on QA Farm | Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007 | Run in customized harness, stable and not necessary to use or share. | bc |
| Security tests | Custom framework. Runs on QA Farm. | Semi-automated. Plans to fully automate with Eggplant in Q1 2007 | When bugs were filed, regression test cases are usually added. | bc | |
| Automated Smoke Tests | A "Smoke Test" is a quick, basic qualification. | Eggplant. Runs on Tracy's machines. Tests for FFx 1.5 and 2.0 | Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move executiion to QA Farm in Q1 2007 | tracy | |
| Basic Functionality Tests (BFT) | 130+ manual test cases, broadly covering major features | litmus for 1.5 and 2.0 | will be updated for FF3 | ||
| Full Functionality Tests (FFT) | approx. 280 manual test cases, covering most features | litmus for 1.0, 1.5 and 2.0 | will be updated for FF3 | ||
| Localization Tests | 15 test cases | litmus for 1.5 and 2.0 | engaged and open to design | axel | |
| Software Update Suite | approx. 50 tests, targeted to update process itself | litmus | |||
| T* | includes Tdhtml, Ts, Txul, Trender (aka Tgfx), Z, and mZ | "make check" | running, but not growing | most T Trender | |
| Tp2 | new page load tests | "make check"/buildbot | nearing complete automation | alice | |
| Download Checker | verify integrity of major released | manual | runnable, passing | wiki | bc |
| L10N Search Verifier | manual | runnable, passing | wiki | bc | |
| XSLT | re-purposing of the Adobe Xalan tests. | custom executable/C++ | Driver is in mozilla, but test content is at Apache. | content/xslt/buster + files downloaded from svn.apache.org content/xslt/XSLTMark - info on this? |
|
| DOM | being transitioned into MochiTest | browser and eyeballs? | dom/tests/js/*html and js | ||
| CSS | verify spec compliance | reftest | running, passing, needing much to be added | dbaron, robcee | |
| XBL2 | XBL2 still in design | wiki | jonas | ||
| Browser/Content | May be transitioned to reftest or MochiTest, as appropriate. | custom executable/C++ | dirs = browser/components/places, browser/components/safebrowsing, browser/components/sessionstore, content/base, content/events, content/html/content, content, content/xml, content/xslt | ||
| Embedding | Tests seem fairly current but not building, why? | custom executable/C++ | browser/activex/src/install/smoketest.html browser/activex/src/plugin/test.htm browser/activex/src/pluginhostctrl/tests browser/activex/tests/cbrowse/*.cpp browser/gtk/tests/*.cpp browser/photon/tests/*.cpp browser/qt/tests/*.cpp qa/jstests/*.txt (for JS, why txt?) qa/testembed/*.cpp tests/apitest/apitest.cpp tests/unit/test_wwpromptfactory.js |
||
| Extensions | custom executable/C++ | dirs = extensions/canvas3d, extensions/java/xpcom, extensions/metrics, extensions/mono, extensions/python, extensions/schema-validation, extensions/spatialnavigation, extensions/webdav | |||
| I18N | custom executable/C++ | dirs = intl, extensions/universalchardet | |||
| Graphics | custom executable/C++ | dir = gfx, tests = nsCoverage, DumpColors, TestColorNames, TestRect | |||
| HTML Parsing | Will need to deal with WHATWG/html5lib tests 366936. The htmlparser/tests/html test vs a baseline, but a baseline has not been stored. | custom executable/C++ | [README] parser/xml/tests/unit/test_parser.js (for xpcshell) parser/htmlparser/robot/test (may be vestigal) parser/htmlparser/tests/outsinks parser/htmlparser/tests/html |
mrbkap | |
| Acid2 | compliance tests from outside standards "watcher" | manual | currently passing | Acid2 Browser Test | |
| Update Checker | validity of update mar files | currently manual in QA Farm, moving to buildbot | wiki | bc | |
| XPConnect | TestXPC -> "FAILED to get a JSRuntime" | custom executable/C++ | dir = js/src/xpconnect | ||
| Layout/HTML | 1800+ files. These can be transitioned, probably to reftest. | "the old regression test suite" and reftest. Older tests will go to reftest. | HTML file groups: CellMapTest, FrameSetNav, SlideShow, attributes, block, css, formctls, frameset, printer, style, table, tags, xbl. | ||
| Math/ML | Do mathematical glyphs display correctly on all platforms? | "browsers and eyeballs" | also using w3c.org site content | dir = layout/mathml | |
| Plugins | Not known | dir = plugin/tools/tester | |||
| Java Plugin | not current, dev may feel out of loop | dirs = modules/libjar, modules/oji | |||
| Networking | 25 test executables, but all must be given parameters | custom executable/C++ | dir = netwerk | ||
| NSPR | custom executable/c | approx 150 test executables, most pass. | |||
| RDF | custom executable/C++ | dsds (not being built), rdfcat, rdfpoll, triplescat (doc for invoke?) | dir = rdf | ||
| Security/NSS | 26 tests, but not runnable without info | custom executable/C++ and "make check" | dir = security/nss | ||
| XPCOM | 28 custom executables, most pass, can be better documented | custom executable/C++ | xpcom/tests/unit/test_pipe.js (xpcshell) | ||
| XPFE | winopen.xul, runs but does not show final results | dir = xpfe | |||
| XPInstall | pre_checkin_trigger.html, runs in browser, passes | dir = xpinstall |
Test Infrastructures
| Test Infrastructure Name | Brief Description | Harnesses and Test Suites in the Infrastructure | Status of the Instructure | Notes |
| tinderboxen | Continuous build engine for Mozilla source, much chronicled in song and fable. | "make check", the results of which may not be verified | see dynamic Tinderbox display | |
| buildbots | Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins. | "make check", reftest and MochiTest | reported via MozillaTest tinderbox | |
| QA Farm | some number of Windows, Linux and Macs. Details are not important. They should be a black box resource for developers to say "test this". | "make check", reftest and MochiTest |