QA/TDAI/Inventory: Difference between revisions
Jump to navigation
Jump to search
(→Test Harnesses: add desc of spindermonkey) |
(→Test Suites: update descriptions) |
||
| Line 119: | Line 119: | ||
|- | |- | ||
| JS tests | | JS tests | ||
| | | well-developed suite of tests of the JavaScript language | ||
| Custom framework. Runs on QA Farm. | | Custom framework. Runs on QA Farm. | ||
| Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007 | | Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007 | ||
| | | Run in customized harness, stable and not necessary to use or share. | ||
| | | [mailto:bc@mozilla.com bc] | ||
|- | |- | ||
| Security tests | | Security tests | ||
| | | well-developed suite security tests | ||
| Custom framework. Runs on QA Farm. | | Custom framework. Runs on QA Farm. | ||
| Semi-automated. Plans to fully automate with Eggplant in Q1 2007 | | Semi-automated. Plans to fully automate with Eggplant in Q1 2007 | ||
| | | Not checked in to repository for security reasons. | ||
| | | bc@mozilla.com | ||
|- | |- | ||
| Automated Smoke Tests - FFx 1.5 | | Automated Smoke Tests - FFx 1.5 | ||
| Line 136: | Line 136: | ||
| Eggplant. Runs on Tracy's machines. | | Eggplant. Runs on Tracy's machines. | ||
| Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move executiion to QA Farm in Q1 2007 | | Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move executiion to QA Farm in Q1 2007 | ||
| | | A "Smoke Test" is a quick, basic qualification. | ||
| | | tracy@mozilla.com | ||
|- | |- | ||
| Automated Smoke Tests - FFx 2.0 | | Automated Smoke Tests - FFx 2.0 | ||
| | | | ||
| Eggplant, runs on Tracy's machines. Originally a litmus suite. | | Eggplant, runs on Tracy's machines. Originally a litmus suite. | ||
| Fully automated. Completed porting 1.5 tests to the 3 platforms for 2.0 in Q4 2006. Plan to move | | Fully automated. Completed porting 1.5 tests to the 3 platforms for 2.0 in Q4 2006. Plan to move execution to QA Farm in Q1 2007 | ||
| | | | ||
| tracy@mozilla.com | |||
|- | |- | ||
| Basic Functionality Tests (BFT) | | Basic Functionality Tests (BFT) | ||
| Line 174: | Line 174: | ||
|- | |- | ||
| Browser/Content | | Browser/Content | ||
| | | Will be transitioned to reftest of MochiTest or reftest as appropriate. | ||
| custom and/or old, not functional | | custom and/or old, not functional | ||
| | | | ||
| | | | ||
browser/components/places=9<br>browser/components/safebrowsing=1<br>browser/components/sessionstore=1<br>content/base=9<br>content/events=0<br>content/html/content=15<br>content=10<br>content/xml=35<br>content/xslt=23<br>content/xslt/XSLTMark=5 | browser/components/places=9<br>browser/components/safebrowsing=1<br>browser/components/sessionstore=1<br>content/base=9<br>content/events=0<br>content/html/content=15<br>content=10<br>content/xml=35<br>content/xslt=23<br>content/xslt/XSLTMark=5 | ||
| | | | ||
| Line 185: | Line 185: | ||
| custom driver in repository | | custom driver in repository | ||
| Driver is in mozilla, but test content gets re-downloaded. | | Driver is in mozilla, but test content gets re-downloaded. | ||
| content/xslt/buster | | content/xslt/buster<br>ToDo: downloadable tests are not easy to find. Where are they? | ||
| | | | ||
|- | |- | ||
| Line 241: | Line 241: | ||
| "the old regression test suite" and reftest. Older tests will go to reftest. | | "the old regression test suite" and reftest. Older tests will go to reftest. | ||
| | | | ||
| | | There are a lot of test files here. It is hard to say how current or correct they are. | ||
| | | | ||
|- | |- | ||
| Math/ML | | Math/ML | ||
| | | Do mathematical glyphs display correctly on all platforms? | ||
| "browsers and eyeballs" | | "browsers and eyeballs" | ||
| also using w3c.org site content | | also using w3c.org site content | ||
Revision as of 18:26, 13 February 2007
We all want quality to happen. How can we help? What is being run? What can I run something with? What does any of this mean? Not all questions are answered, but some.
This document has three sections:
- Test Harnesses - tools, designed to run tests. A harness may be targeted to a specific set of tests, or general to many kinds of tests. What test do you want to see run? Different harnesses could be used to run a given test. Which harness is better for which kinds of testing? kinds of testing?
- Test Suites - arbitrary groupings of test cases, roughly organized by functional area
- Test Infrastructure - the environments the test tools and test cases run in. This includes machine, operating system and physical location (ex: QA farm vs QA lab vs someone's desktop)
|
THIS PAGE IS A WORKING DRAFT | 
|
| The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page. |
Test Harnesses
| Test Harness Name | Brief Description | Language Used | Status of Harness | Notes |
| MochiTest | based on the MochiKit framework | JavaScript | Invocation by buildbot on QA machines is being set up. Invocation of tests by inidividuals on built trunk is possible with setup. Currently being made to work in automated test harnesses. | [MochiTest on devmo] |
| jsunit | JavaScript | |||
| Acid2 | Web Standards Project compliance test page(s) | [Stds page] | ||
| litmus | Catalog of manual test cases | English, German, French, etc. | The place to be on Community Test Day! | [main site] |
| xpcshell | lots of under-the-hood XPCOM smarts! | some JavaScript, much C++ | [xpcshell on devmo] | |
| SpiderMonkey | used only for the js/tests by bc, et al, for JS testing | JavaScript | [bc's page] | |
| reftest | comparison of two pages, via JavaScript and Canvas, for visual equivalency | JavaScript | runnable on built trunk | [reftest on devmo] |
| jssh | a javascript shell. What more can you say? | JavaScript | described as "largely defunct" | |
| Tp | test of page loading | currently runs in tinderbox | ||
| Tp2 | test of page loading | being developed to run in buildbot | ||
| T* | includes Tdhtml, Ts, Txul, Trender (aka Tgfx), Z, and mZ | [most T] [Trender] | ||
| 'Browser and Eyeballs' | The most common test harness in use today. | wetware | There is a new tester born every day! :-) | |
| Download Checker | listed as "run manually by QA" ??? | [wiki page] | ||
| Software Update Checker | listed as "run manually by QA" ??? | [wiki page] | ||
| l10n Search Verifier | listed as "run manually by QA" ??? | [wiki page] |
Test Suites
| Test Suite Name | Brief Description | Test Harness(es) | Status | Notes | Owner |
| JS tests | well-developed suite of tests of the JavaScript language | Custom framework. Runs on QA Farm. | Semi-automated. Plans to fully automate with Alice and Rob's help in Q1 2007 | Run in customized harness, stable and not necessary to use or share. | bc |
| Security tests | well-developed suite security tests | Custom framework. Runs on QA Farm. | Semi-automated. Plans to fully automate with Eggplant in Q1 2007 | Not checked in to repository for security reasons. | bc@mozilla.com |
| Automated Smoke Tests - FFx 1.5 | Eggplant. Runs on Tracy's machines. | Fully automated. Completed porting to Mac and Linux in Q4 2006. Plan to move executiion to QA Farm in Q1 2007 | A "Smoke Test" is a quick, basic qualification. | tracy@mozilla.com | |
| Automated Smoke Tests - FFx 2.0 | Eggplant, runs on Tracy's machines. Originally a litmus suite. | Fully automated. Completed porting 1.5 tests to the 3 platforms for 2.0 in Q4 2006. Plan to move execution to QA Farm in Q1 2007 | tracy@mozilla.com | ||
| Basic Functionality Tests (BFT) | in http://litmus.mozilla.org | ||||
| Full Functionality Tests (FFT) | in http://litmus.mozilla.org | ||||
| Localization Tests | in http://litmus.mozilla.org | ||||
| Software Update Suite | in Litmus | ||||
| Browser/Content | Will be transitioned to reftest of MochiTest or reftest as appropriate. | custom and/or old, not functional |
browser/components/places=9 |
||
| Buster | re-purposing of the Adobe Xalan tests. | custom driver in repository | Driver is in mozilla, but test content gets re-downloaded. | content/xslt/buster ToDo: downloadable tests are not easy to find. Where are they? |
|
| DOM | browser and eyeballs? | dom/tests/html=1 dom/tests/js=19 |
|||
| Embedding | ??? | embedding/browser/activex=87 embedding/browser/gtk=5 embedding/browser/photon=1 embedding/browser/qt=7 embedding/qa/jstests=88 embedding/qa/jstests/accessibility=53 embedding/qaembed=55 embedding/qaembed=55 embedding/apitest=2 embedding=113 |
|||
| Extensions | ??? | extensions/canvas3d=6 extensions/java/xpcom=10 extensions/metrics=5 extensions/mono=5 extensions/python=31 extensions/schema-validation=9 extensions/spatialnavigation=5 extensions/universalchardet=2 extensions/webdav=1 |
|||
| Graphics | custom executable/C++ | gfx=8 | |||
| HTML Parsing | custom harness (non-functioning). Moving to WHATWG/html5lib tests [366936] | [README] parser/htmlparser=327 |
Blake Kaplan | ||
| I18N | custom executable/C++ | intl=37 | |||
| XPConnect | custom executable/C++ | js/src/xpconnect=142 | |||
| Layout/HTML | "the old regression test suite" and reftest. Older tests will go to reftest. | There are a lot of test files here. It is hard to say how current or correct they are. | |||
| Math/ML | Do mathematical glyphs display correctly on all platforms? | "browsers and eyeballs" | also using w3c.org site content | layout/mathml=7 | |
| ??? | modules/libjar=4 modules/oji=758 modules/plugin=63 |
||||
| Networking | custom executable/C++ | netwerk=121 | |||
| NSPR | custom executable/C | nsprpub/lib=3 | |||
| RDF | custom executable/C++ | rdf=11 | |||
| Security/NSS | custom executable and "make check" | security/dbm=0 security/nss=56 |
|||
| XPCOM | custom executable/C++ | xpcom/proxy=6 xpcom=66 |
|||
| XPFE | JavaScript/XULRunner? | xpfe=3 | |||
| XPInstall | ??? | xpinstall=3 |
Test Infrastructures
| Test Infrastructure Name | Brief Description | Harnesses and Test Suites in the Infrastructure | Status of the Instructure | Notes |
| tinderboxen | Continuous build engine for Mozilla source, much chronicled in song and fable. | "make check", the results of which may not be verified | see dynamic Tinderboxen display | |
| buildbots | Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins. Running make check, reftest and mochitest | |||
| QA Farm | some number of Windows, Linux and Macs. Details are not important. They should be a black box resource for developers to say "test this". |