QA/TDAI/Inventory

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

We all want quality to happen. How can we help? What is being run? What can I run something with? What does any of this mean? Not all questions are answered, but some.

This document has three sections:

1. Test Harnesses - tools, designed to run tests. A harness may be targeted to a specific set of tests, or general to many kinds of tests. What test do you want to see run? Different harnesses could be used to run a given test. Which harness is better for which kinds of testing? kinds of testing?
2. Test Suites - arbitrary groupings of test cases, roughly organized by functional area
3. Test Infrastructure - the environments the test tools and test cases run in. This includes machine, operating system and physical location (ex: QA farm vs QA lab vs someone's desktop)

There are questions to be answered about any of our tests. If anyone has information or suggestions, please let us know.

1. What tests do we have now, how current are they and how they should be run? Should they be expected to pass?
2. What features or modules need tests to be written for them? What are the requirements of the testing that should happen?
3. What tools do we have now that can be used to write tests? What tools outside of mozilla have people used to build tests and test tools.
4. Of the tests currently in existence, which of them are outside the "mozilla.org" umbrella? Which use resources from outside? What is the reliability and accessibility of these resources? Are there issues which keep us from keeping local copies?

Any questions and suggestions are welcome!

Test Harnesses

Test Harness Name	Brief Description	Language Used	Status of Harness	Notes
MochiTest	built by sayrer using the MochiKit framework	JavaScript	Automation being set up on QA Farm. Exec by community is possible with setup.	devmo
eggplant		uses VNC for device-level visual comparisons	Being used for automation of Smoke Tests suites in litmus	redstone
litmus	Catalog of manual test cases. Being developed to allow community to run the manual smoke tests, BFT, FFT and L10n tests. GUI to capture results and provide feedback to the QA team. Reporting function to check test and platform coverage.	English, German, French, etc.	The place to be on Community Test Day! Still in refinement by zlipton and coop. 2445 test cases, 1994 tester accounts with 845 active	litmus
T*	perf tests, runs in tinderbox		running, passing, run by tinderbox	wiki
xpcshell	used by 40+ tests, plans for use by many more	JavaScript	recent care and feeding, very usable	devmo
Spider	crawls a site to specified depth, loading CSS, executing JS along the way.	JavaScript	used only for the js/tests by bc, et al, for JS testing	bc
reftest	comparison of two pages, via JavaScript and Canvas, for visual equivalency	JavaScript	runnable on built trunk	devmo
build/make	a standard for invoking stand-alone test executables, and for integrating invocation of other harnesses into MoCo automated systems		"make check" is accepted standard, being integrated with buildbot	tinderbox

Test Suites

Test Suite Name	Brief Description	Test Harness(es)	Status	Notes	Owner
JS tests	well-developed suite of tests of the JavaScript language in js/tests	SpiderMonkey Runs on QA Farm	Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007	Run in customized harness, stable and not necessary to use or share.	bc
Security tests		Custom framework. Runs on QA Farm.	Semi-automated. Plans to fully automate with Eggplant in Q1 2007	When bugs were filed, regression test cases are usually added.	bc
Automated Smoke Tests	A "Smoke Test" is a quick, basic qualification.	Eggplant. Runs on Tracy's machines. Tests for FFx 1.5 and 2.0	Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move executiion to QA Farm in Q1 2007		tracy
Basic Functionality Tests (BFT)	130+ manual test cases, broadly covering major features	litmus for 1.5 and 2.0	will be updated for FF3
Full Functionality Tests (FFT)	approx. 280 manual test cases, covering most features	litmus for 1.0, 1.5 and 2.0	will be updated for FF3
Localization Tests	15 test cases	litmus for 1.5 and 2.0	engaged and open to design		axel
Software Update Suite	approx. 50 tests, targeted to update process itself	litmus
T*	includes Tdhtml, Ts, Txul, Trender (aka Tgfx), Z, and mZ	"make check"	running, but not growing	most T Trender
Tp2	new page load tests	"make check"/buildbot	nearing complete automation		alice
Download Checker	verify integrity of major released	manual	runnable, passing	wiki	bc
L10N Search Verifier		manual	runnable, passing	wiki	bc
XSLT	re-purposing of the Adobe Xalan tests.	custom executable/C++	Driver is in mozilla, but test content is at Apache.	content/xslt/buster + files downloaded from svn.apache.org content/xslt/XSLTMark - info on this?
DOM	being transitioned into MochiTest	browser and eyeballs?		dom/tests/js/*html and js
CSS	verify spec compliance	reftest	running, passing, needing much to be added		dbaron, robcee
XBL2			XBL2 still in design	wiki	jonas
Browser/Content	May be transitioned to reftest or MochiTest, as appropriate.	custom executable/C++		dirs = browser/components/places, browser/components/safebrowsing, browser/components/sessionstore, content/base, content/events, content/html/content, content, content/xml, content/xslt
Embedding	Tests seem fairly current but not building, why?	custom executable/C++		browser/activex/src/install/smoketest.html browser/activex/src/plugin/test.htm browser/activex/src/pluginhostctrl/tests browser/activex/tests/cbrowse/*.cpp browser/gtk/tests/*.cpp browser/photon/tests/*.cpp browser/qt/tests/*.cpp qa/jstests/*.txt (for JS, why txt?) qa/testembed/*.cpp tests/apitest/apitest.cpp tests/unit/test_wwpromptfactory.js
Extensions		custom executable/C++		dirs = extensions/canvas3d, extensions/java/xpcom, extensions/metrics, extensions/mono, extensions/python, extensions/schema-validation, extensions/spatialnavigation, extensions/webdav
Java Applets	list suggested by Java team at Apple	browser and eyeballs	Just a list. No automation is currently in process.	wiki	rkiddy
I18N		custom executable/C++		dirs = intl, extensions/universalchardet
Graphics		custom executable/C++		dir = gfx, tests = nsCoverage, DumpColors, TestColorNames, TestRect
HTML Parsing	Will need to deal with WHATWG/html5lib tests 366936. The htmlparser/tests/html test vs a baseline, but a baseline has not been stored.	custom executable/C++		[README] parser/xml/tests/unit/test_parser.js (for xpcshell) parser/htmlparser/robot/test (may be vestigal) parser/htmlparser/tests/outsinks parser/htmlparser/tests/html	mrbkap
Acid2	compliance tests from outside standards "watcher"	manual	currently passing	Acid2 Browser Test
Update Checker	validity of update mar files	currently manual in QA Farm, moving to buildbot		wiki	bc
XPConnect	TestXPC -> "FAILED to get a JSRuntime"	custom executable/C++		dir = js/src/xpconnect
Layout/HTML	1800+ files. These can be transitioned, probably to reftest.	"the old regression test suite" and reftest. Older tests will go to reftest.		HTML file groups: CellMapTest, FrameSetNav, SlideShow, attributes, block, css, formctls, frameset, printer, style, table, tags, xbl.
Math/ML	Do mathematical glyphs display correctly on all platforms?	"browsers and eyeballs"	also using w3c.org site content	dir = layout/mathml
Plugins			Not known	dir = plugin/tools/tester
Java Plugin			not current, dev may feel out of loop	dirs = modules/libjar, modules/oji
Networking	25 test executables, but all must be given parameters	custom executable/C++		dir = netwerk
NSPR		custom executable/c		approx 150 test executables, most pass.
RDF		custom executable/C++	dsds (not being built), rdfcat, rdfpoll, triplescat (doc for invoke?)	dir = rdf
Security/NSS	26 tests, but not runnable without info	custom executable/C++ and "make check"		dir = security/nss
XPCOM	28 custom executables, most pass, can be better documented	custom executable/C++		xpcom/tests/unit/test_pipe.js (xpcshell)
XPFE	winopen.xul, runs but does not show final results			dir = xpfe
XPInstall	pre_checkin_trigger.html, runs in browser, passes			dir = xpinstall

Test Infrastructures

Test Infrastructure Name	Brief Description	Harnesses and Test Suites in the Infrastructure	Status of the Instructure	Notes
tinderboxen	Continuous build engine for Mozilla source, much chronicled in song and fable.	"make check", the results of which may not be verified	see dynamic Tinderbox display
buildbots	Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins.	"make check", reftest and MochiTest	reported via MozillaTest tinderbox
QA Farm	some number of Windows, Linux and Macs. Details are not important. They should be a black box resource for developers to say "test this".	"make check", reftest and MochiTest