QA/TDAI/Inventory

We all want quality to happen. How can we help? What is being run? What can I run something with? What does any of this mean? Not all questions are answered, but some.

This document has three sections:

1. Test Harnesses - tools, designed to run tests. A harness may be targeted to a specific set of tests, or general to many kinds of tests. What test do you want to see run? Different harnesses could be used to run a given test. Which harness is better for which kinds of testing? kinds of testing?
2. Test Suites - arbitrary groupings of test cases, roughly organized by functional area
3. Test Infrastructure - the environments the test tools and test cases run in. This includes machine, operating system and physical location (ex: QA farm vs QA lab vs someone's desktop)

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

Test Harnesses

Test Harness Name	Brief Description	Language Used	Status of Harness	Notes
MochiTest	based on the MochiKit framework	JavaScript	Invocation by buildbot on QA machines is being set up. Invocation of tests by inidividuals on built trunk is possible with setup. Currently being made to work in automated test harnesses.	[MochiTest on devmo]
jsunit		JavaScript
Acid2	Web Standards Project compliance test page(s)			[Stds page]
litmus	Catalog of manual test cases	English, German, French, etc.	The place to be on Community Test Day!	[main site]
xpcshell	lots of under-the-hood XPCOM smarts!	some JavaScript, much C++		[xpcshell on devmo]
SpiderMonkey	used only for the js/tests by bc, et al, for JS testing	JavaScript		[bc's page]
reftest	comparison of two pages, via JavaScript and Canvas, for visual equivalency	JavaScript	runnable on built trunk	[reftest on devmo]
jssh	a javascript shell. What more can you say?	JavaScript	described as "largely defunct"
Tp	test of page loading		currently runs in tinderbox
Tp2	test of page loading		being developed to run in buildbot
T*	includes Tdhtml, Ts, Txul, Trender (aka Tgfx), Z, and mZ			[most T] [Trender]
'Browser and Eyeballs'	The most common test harness in use today.	wetware	There is a new tester born every day! :-)
Download Checker			listed as "run manually by QA" ???	[wiki page]
Software Update Checker			listed as "run manually by QA" ???	[wiki page]
l10n Search Verifier			listed as "run manually by QA" ???	[wiki page]

Test Suites

Test Suite Name	Brief Description	Test Harness(es)	Status	Notes	Owner
JS tests		Custom framework. Runs on QA Farm.	Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007
Security tests		Custom framework. Runs on QA Farm.	Semi-automated. Plans to fully automate with Eggplant in Q1 2007
Automated Smoke Tests - FFx 1.5		Eggplant. Runs on Tracy's machines.	Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move executiion to QA Farm in Q1 2007
Automated Smoke Tests - FFx 2.0		Eggplant, runs on Tracy's machines. Originally a litmus suite.	Fully automated. Completed porting 1.5 tests to the 3 platforms for 2.0 in Q4 2006. Plan to move executiion to QA Farm in Q1 2007
Basic Functionality Tests (BFT)		in http://litmus.mozilla.org
Full Functionality Tests (FFT)		in http://litmus.mozilla.org
Localization Tests		in http://litmus.mozilla.org
Software Update Suite		in Litmus
Browser/Content		custom and/or old, not functional		Will be transitioned to reftest of MochiTest or reftest as appropriate. browser/components/places=9 browser/components/safebrowsing=1 browser/components/sessionstore=1 content/base=9 content/events=0 content/html/content=15 content=10 content/xml=35 content/xslt=23 content/xslt/XSLTMark=5
Buster	re-purposing of the Adobe Xalan tests.	custom driver in repository	Driver is in mozilla, but test content gets re-downloaded.	content/xslt/buster
DOM		browser and eyeballs?		dom/tests/html=1 dom/tests/js=19
Embedding		???		embedding/browser/activex=87 embedding/browser/gtk=5 embedding/browser/photon=1 embedding/browser/qt=7 embedding/qa/jstests=88 embedding/qa/jstests/accessibility=53 embedding/qaembed=55 embedding/qaembed=55 embedding/apitest=2 embedding=113
Extensions		???		extensions/canvas3d=6 extensions/java/xpcom=10 extensions/metrics=5 extensions/mono=5 extensions/python=31 extensions/schema-validation=9 extensions/spatialnavigation=5 extensions/universalchardet=2 extensions/webdav=1
Graphics		custom executable/C++		gfx=8
HTML Parsing		custom harness (non-functioning). Moving to WHATWG/html5lib tests [366936]		[README] parser/htmlparser=327	Blake Kaplan
I18N		custom executable/C++		intl=37
XPConnect		custom executable/C++		js/src/xpconnect=142
Layout/HTML		"the old regression test suite" and reftest. Older tests will go to reftest.		layout/html=2461 layout/reftests=360 layout/svg=2 layout/tools/reftest=6 layout/tools=4
Math/ML		"browsers and eyeballs"	also using w3c.org site content	layout/mathml=7
???				modules/libjar=4 modules/oji=758 modules/plugin=63
Networking		custom executable/C++		netwerk=121
NSPR		custom executable/C		nsprpub/lib=3
RDF		custom executable/C++		rdf=11
Security/NSS		custom executable and "make check"		security/dbm=0 security/nss=56
XPCOM		custom executable/C++		xpcom/proxy=6 xpcom=66
XPFE		JavaScript/XULRunner?		xpfe=3
XPInstall		???		xpinstall=3

Test Infrastructures

Test Infrastructure Name	Brief Description	Harnesses and Test Suites in the Infrastructure	Status of the Instructure	Notes
tinderboxen	Continuous build engine for Mozilla source, much chronicled in song and fable.	"make check", the results of which may not be verified	see dynamic Tinderboxen display
buildbots	Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins. Running make check, reftest and mochitest
QA Farm	some number of Windows, Linux and Macs. Details are not important. They should be a black box resource for developers to say "test this".