QA/TDAI/Inventory: Difference between revisions

From MozillaWiki
< QA‎ | TDAI
Jump to navigation Jump to search
(→‎Test Suites: adding info)
(→‎Test Suites: adding litmus name)
Line 123: Line 123:
| Basic Functionality Tests (BFT)
| Basic Functionality Tests (BFT)
| 130+ manual test cases, broadly covering major features
| 130+ manual test cases, broadly covering major features
| for [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=7 1.5] and [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=25 2.0]
| litmus for [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=7 1.5] and [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=25 2.0]
|
|
|  
|  
Line 130: Line 130:
| Full Functionality Tests (FFT)
| Full Functionality Tests (FFT)
| approx. 280 manual test cases, covering most features
| approx. 280 manual test cases, covering most features
| for [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=2 1.0], [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=26 1.5] and [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=27 2.0]
| litmus for [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=2 1.0], [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=26 1.5] and [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=27 2.0]
|
|
|  
|  
Line 137: Line 137:
| Localization Tests
| Localization Tests
| 15 test cases
| 15 test cases
| for [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=10 1.5] and [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=39 2.0]
| litmus for [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=10 1.5] and [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&testgroup_id=39 2.0]
|
|
|  
|  

Revision as of 08:01, 14 February 2007

We all want quality to happen. How can we help? What is being run? What can I run something with? What does any of this mean? Not all questions are answered, but some.

This document has three sections:

  1. Test Harnesses - tools, designed to run tests. A harness may be targeted to a specific set of tests, or general to many kinds of tests. What test do you want to see run? Different harnesses could be used to run a given test. Which harness is better for which kinds of testing? kinds of testing?
  2. Test Suites - arbitrary groupings of test cases, roughly organized by functional area
  3. Test Infrastructure - the environments the test tools and test cases run in. This includes machine, operating system and physical location (ex: QA farm vs QA lab vs someone's desktop)
Draft-template-image.png THIS PAGE IS A WORKING DRAFT Pencil-emoji U270F-gray.png
The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly.
If you have any questions or ideas, please add them as a new topic on the discussion page.

Test Harnesses

Test Harness Name Brief Description Language Used Status of Harness Notes
MochiTest based on the MochiKit framework JavaScript Invocation by buildbot on QA machines is being set up. Invocation of tests by inidividuals on built trunk is possible with setup. Currently being made to work in automated test harnesses. MochiTest on devmo
jsunit JavaScript
Acid2 Web Standards Project compliance test page(s) Stds page
litmus Catalog of manual test cases English, German, French, etc. The place to be on Community Test Day! main site
xpcshell lots of under-the-hood XPCOM smarts! some JavaScript, much C++ xpcshell on devmo
SpiderMonkey used only for the js/tests by bc, et al, for JS testing JavaScript bc's page
reftest comparison of two pages, via JavaScript and Canvas, for visual equivalency JavaScript runnable on built trunk reftest on devmo
jssh a javascript shell. What more can you say? JavaScript described as "largely defunct"
Tp test of page loading currently runs in tinderbox
Tp2 test of page loading being developed to run in buildbot
T* includes Tdhtml, Ts, Txul, Trender (aka Tgfx), Z, and mZ most T Trender
'Browser and Eyeballs' The most common test harness in use today. wetware There is a new tester born every day! :-)

Test Suites

Test Suite Name Brief Description Test Harness(es) Status Notes Owner
JS tests well-developed suite of tests of the JavaScript language Custom framework in js/tests. Runs on QA Farm. Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007 Run in customized harness, stable and not necessary to use or share. bc
Security tests Custom framework. Runs on QA Farm. Semi-automated. Plans to fully automate with Eggplant in Q1 2007 When bugs were filed, regression test cases are usually added. bc@mozilla.com
Automated Smoke Tests A "Smoke Test" is a quick, basic qualification. Eggplant. Runs on Tracy's machines. Tests for FFx 1.5 and 2.0 Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move executiion to QA Farm in Q1 2007 tracy@mozilla.com
Basic Functionality Tests (BFT) 130+ manual test cases, broadly covering major features litmus for 1.5 and 2.0
Full Functionality Tests (FFT) approx. 280 manual test cases, covering most features litmus for 1.0, 1.5 and 2.0
Localization Tests 15 test cases litmus for 1.5 and 2.0
Software Update Suite approx. 50 tests, targeted to update process itself litmus
Browser/Content May be transitioned to reftest or MochiTest, as appropriate. custom executable/C++ (I need to break this category up. Too much is happening in here. -rrk)

browser/components/places/testsbrowser/components/places=9
browser/components/safebrowsing=1
browser/components/sessionstore=1
content/base=9
content/events=0
content/html/content=15
content=10
content/xml=35
content/xslt=23
content/xslt/XSLTMark=5

Buster re-purposing of the Adobe Xalan tests. custom executable/C++ Driver is in mozilla, but test content is at Apache. content/xslt/buster + files downloaded from svn.apache.org
DOM browser and eyeballs? dom/tests/js/*html and js, can be made into reftests or MochiTest
Embedding custom executable/C++
Extensions custom executable/C++ extensions/canvas3d
extensions/java/xpcom
extensions/metrics
extensions/mono
extensions/python
extensions/schema-validation
extensions/spatialnavigation
extensions/webdav
Graphics custom executable/C++ gfx
HTML Parsing custom executable/C++ (non-functioning). Moving to WHATWG/html5lib tests [366936] [README]
parser/htmlparser=327
Blake Kaplan
I18N custom executable/C++ intl/, extensions/universalchardet/
XPConnect custom executable/C++ js/src/xpconnect
Layout/HTML "the old regression test suite" and reftest. Older tests will go to reftest. There are a lot of test files here. It is hard to say how current or correct they are.
Math/ML Do mathematical glyphs display correctly on all platforms? "browsers and eyeballs" also using w3c.org site content layout/mathml
??? modules/libjar
modules/oji
modules/plugin
Networking custom executable/C++ netwerk
NSPR custom executable/c approx 150 test executables, most pass.
RDF custom executable/C++ rdf
Security/NSS custom executable/C++ and "make check" security/dbm
security/nss
XPCOM custom executable/C++ xpcom/proxy
xpcom
XPFE xpfe
XPInstall xpinstall

Test Infrastructures

Test Infrastructure Name Brief Description Harnesses and Test Suites in the Infrastructure Status of the Instructure Notes
tinderboxen Continuous build engine for Mozilla source, much chronicled in song and fable. "make check", the results of which may not be verified see dynamic Tinderbox display
buildbots Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins. "make check", reftest and MochiTest reported via MozillaTest tinderbox
QA Farm some number of Windows, Linux and Macs. Details are not important. They should be a black box resource for developers to say "test this". "make check", reftest and MochiTest