MozillaWiki

QA/TDAI/Inventory: Difference between revisions

Page Discussion

QA/TDAI/Inventory (view source)

Revision as of 23:30, 23 April 2010

981 bytes added ,  23 April 2010
m
→‎Suites? Harnesses? Tests?: correct punctuation
(update with accessibility info)
m (→‎Suites? Harnesses? Tests?: correct punctuation)
 
(17 intermediate revisions by 6 users not shown)
Line 1: Line 1:
<small>[[QA/TDAI|&laquo; QA/TDAI]]</small>
{{draft}}
{{draft}}


Line 15: Line 17:
# What tests do we have now, how current are they and how they should be run? Should they be expected to pass?
# What tests do we have now, how current are they and how they should be run? Should they be expected to pass?
# What features or modules need tests to be written for them? What are the requirements of the testing that should happen?
# What features or modules need tests to be written for them? What are the requirements of the testing that should happen?
# What tools do we have now that can be used to write tests? What tools outside of mozilla have people used to build tests and test tools.
# What tools do we have now that can be used to write tests? What tools outside of mozilla have people used to build tests and test tools?
# Of the tests currently in existence, which of them are outside the "mozilla.org" umbrella? Which use resources from outside? What is the reliability and accessibility of these resources? Are there issues which keep us  from keeping local copies?
# Of the tests currently in existence, which of them are outside the "mozilla.org" umbrella? Which use resources from outside? What is the reliability and accessibility of these resources? Are there issues which keep us  from keeping local copies?


Line 22: Line 24:
=== Criteria for Colors ===
=== Criteria for Colors ===


<i>Note that judgements are subjective and not "hard and fast". Caring enough to discuss will change a rating. Not caring enough to discuss is explanation enough.</i>
<div class="note">'''NOTE''': that judgments are subjective and not "hard and fast". Caring enough to discuss will change a rating. Not caring enough to discuss is explanation enough.</div>


* Green: Generously defined now. Info is current or findable. Something is buildable and runnable by someone. There may still be obvious improvements needed.
* Green: Generously defined now. Info is current or findable. Something is buildable and runnable by someone. There may still be obvious improvements needed.
Line 31: Line 33:


{| class="fullwidth-table"
{| class="fullwidth-table"
| style="background:#efefef" | '''Test Harness Name'''
!Test Harness Name
| style="background:#efefef" | '''Brief Description'''
!Brief Description
| style="background:#efefef" | '''Language Used'''
!Language Used
| style="background:#efefef" | '''Status of Harness'''
!Status of Harness
| style="background:#efefef" | '''Notes'''
!Notes
|-  
|-  
| MochiTest
| MochiTest
| built by [mailto:sayrer@gmail.com sayrer] using the [http://www.mochikit.com MochiKit] framework
| built by [mailto:sayrer@gmail.com sayrer] using the [http://www.mochikit.com MochiKit] framework
| JavaScript
| JavaScript
| Automation being set up on QA Farm. Exec by community is possible with setup.
| Automation being set up on QA Farm. Exec by community is possible.
| [http://developer.mozilla.org/en/docs/Writing_MochiTest-based_unit_tests mochitest]
| [[mdc:Writing MochiTest-based unit tests|mochitest]]
|-
|-
| reftest
| reftest
| comparison of two pages, via JavaScript and Canvas, for visual equivalency
| comparison of two pages, via JavaScript and Canvas, for visual equivalency
| JavaScript
| JavaScript
| runnable on built trunk
| runnable on built trunk. There is a list of [[mdc:reftest opportunities files|reftest opportunities]], but it is not clear if we want to use them.
| [http://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests reftest]
| [[mdc:Creating reftest-based unit tests|reftest]]
|-
|-
| xpcshell
| xpcshell
Line 53: Line 55:
| JavaScript
| JavaScript
| recent care and feeding, very usable
| recent care and feeding, very usable
| [http://developer.mozilla.org/en/docs/Writing_xpcshell-based_unit_tests xpcshell]
| [[mdc:Writing_xpcshell-based_unit_tests|xpcshell]]
|-
|-
| eggplant
| eggplant
Line 62: Line 64:
|-
|-
| litmus
| litmus
| Catalog of manual test cases. [http://wiki.mozilla.org/Litmus Being developed] to allow community to [http://litmus.mozilla.org/run_tests.cgi run] the manual smoke tests, BFT, FFT and L10n tests. [http://litmus.mozilla.org/search_results.cgi GUI to capture results] and provide feedback to the QA team.  Reporting function to check test and platform coverage.
| Catalog of manual test cases. [[Litmus|Being developed]] to allow community to [http://litmus.mozilla.org/run_tests.cgi run] the manual smoke tests, BFT, FFT and L10n tests. [http://litmus.mozilla.org/search_results.cgi GUI to capture results] and provide feedback to the QA team.  Reporting function to check test and platform coverage.
| English, German, French, etc.
| English, German, French, etc.
| The place to be on Community Test Day! Still in refinement by [mailto:zlipton@mozilla.com zlipton] and [mailto:ccooper@deadsquid.com coop]. 2445 test cases, 1994 tester accounts with 845 active  
| The place to be on Community Test Day! Still in refinement by [mailto:zlipton@mozilla.com zlipton] and [mailto:ccooper@deadsquid.com coop]. 2445 test cases, 1994 tester accounts with 845 active  
Line 71: Line 73:
|
|
| running, passing, run by tinderbox
| running, passing, run by tinderbox
| [http://wiki.mozilla.org/MozillaQualityAssurance:Performance_Testing wiki]
| [[MozillaQualityAssurance:Performance_Testing|wiki]]
|-
|-
| test.js
| test.js
Line 96: Line 98:
| any info known? please [mailto:rkiddy@mozilla.com share].
| any info known? please [mailto:rkiddy@mozilla.com share].
| [http://xulunit.mozdev.org xulunit.m.o]
| [http://xulunit.mozdev.org xulunit.m.o]
|-
|litmus2
|redesign/rework of the test case management tool.
|
|in idea phase
|[https://wiki.mozilla.org/index.php?title=QA/TDAI/Inventory:notes notes]
|}
|}


Line 101: Line 109:


{| class="fullwidth-table"
{| class="fullwidth-table"
| style="background:#efefef" |
!
| style="background:#efefef" | '''Test Suite Name'''
!Test Suite Name
| style="background:#efefef" | '''Brief Description'''
!Brief Description
| style="background:#efefef" | '''Test Harness(es)'''
!Test Harness(es)
| style="background:#efefef" | '''Status'''
!Status
| style="background:#efefef" | '''Last Info'''
!Last Info
|-  
|-  
| style="background:#00ff00" |
| style="background:#00ff00" |
| JS tests
| [http://lxr.mozilla.org/mozilla/source/js/tests/ js/tests]
| well-developed suite of tests of the JavaScript language in [http://lxr.mozilla.org/mozilla/source/js/tests/ js/tests]
| [http://www.mozilla.org/js/tests/library.html well-developed suite] of tests of the JavaScript language in [http://lxr.mozilla.org/mozilla/source/js/tests/ js/tests]
| [http://bclary.com/projects/spider/ SpiderMonkey] Runs on QA Farm. Stable and not necessary to use or share.
| [http://bclary.com/projects/spider/ SpiderMonkey] Runs on QA Farm. Stable and not necessary to use or share.
| Semi-automated.  Plans to fully automate with Alice and Rob's help in Q1 2007
| Semi-automated.  Plans to fully automate with Alice and Rob's help in Q1 2007
Line 159: Line 167:
| style="background:#00ff00" |
| style="background:#00ff00" |
| T*
| T*
| includes Tdhtml, Ts, Txul, [http://wiki.mozilla.org/Mozilla2:Trender Trender] (aka Tgfx), Z, and mZ, [http://wiki.mozilla.org/Performance:Tinderbox_Tests wiki for most T]
| includes Tdhtml, Ts, Txul, [[Trender]] (aka Tgfx), Z, and mZ, [[Performance:Tinderbox_Tests|wiki for most T]]
| "make check"
|
| running, but not growing
| running, but not growing
| Info is current. Tests need transition to buildbot. See [mailto:rcampbell@mozilla.com robcee], [mailto:timr@mozilla.com timr].
| Info is current. Tests need transition to buildbot. See [mailto:rcampbell@mozilla.com robcee], [mailto:timr@mozilla.com timr].
Line 166: Line 174:
| style="background:#00ff00" |
| style="background:#00ff00" |
| Accessibility
| Accessibility
| [http://wiki.mozilla.org/Accessibility/qa http://wiki.mozilla.org/Accessibility/qa]
| [[Accessibility/qa]]
|  
|  
| Info is current. A lot is going on, but opportunities to automate are not as developed as they could be. See [mailto:aaronleventhal@moonset.net Aaron Leventhal].
| Info is current. A lot is going on, but opportunities to automate are not as developed as they could be. See [mailto:aaronleventhal@moonset.net Aaron Leventhal].
Line 174: Line 182:
| Update Checker
| Update Checker
| validity of update mar files
| validity of update mar files
| currently manual in QA Farm, moving to buildbot, [http://wiki.mozilla.org/MozillaQualityAssurance:Update_Checker wiki]
| currently automated on release machines, moving to buildbot, [[MozillaQualityAssurance:Update_Checker|wiki]]
|  
|  
| Info is current. See [mailto:bc@mozilla.com bc], [mailto:rhelmer@mozilla.com rhelmer].
| Info is out of date. See [mailto:rhelmer@mozilla.com rhelmer].
|-
| style="background:#00ff00" |
| L10n checker/metadiff
| verify that :<ul><li>en-US build matches locale build</li><li>ensure no read-only files</li><li>show differences from last release</li>
|
| currently automated on release machines, moving to buildbot, [[MozillaQualityAssurance:MetaDiff|wiki]]
| Info is out of date. See [mailto:rhelmer@mozilla.com rhelmer].
|-
|-
| style="background:#00ff00" |
| style="background:#00ff00" |
Line 184: Line 199:
| nearing complete automation
| nearing complete automation
| Info is current. Being integrated into automation. See [mailto:alice@mozilla.com alice], [mailto:rcampbell@mozilla.com robcee].
| Info is current. Being integrated into automation. See [mailto:alice@mozilla.com alice], [mailto:rcampbell@mozilla.com robcee].
|-
| style="background:#00ff00" |
| Update Checker
| validity of update mar files
| currently manual in QA Farm, moving to buildbot, [http://wiki.mozilla.org/MozillaQualityAssurance:Update_Checker wiki]
|
| Info is current. See [mailto:bc@mozilla.com bc], [mailto:rhelmer@mozilla.com rhelmer].
|-
|-
| style="background:#00ff00" |
| style="background:#00ff00" |
| Download Checker
| Download Checker
| verify integrity of major released
| verify integrity of major released
| manual, [http://wiki.mozilla.org/MozillaQualityAssurance:Download_Checker wiki]
| manual, [[MozillaQualityAssurance:Download_Checker|wiki]]
| runnable, passing
| runnable, passing
| Info is current. See [mailto:bc@mozilla.com bc].
| Info is current. See [mailto:bc@mozilla.com bc].
Line 202: Line 210:
| L10N Search Verifier
| L10N Search Verifier
|
|
| manual, [http://wiki.mozilla.org/MozillaQualityAssurance:l10n_Search_Verifier wiki]
| manual, [[MozillaQualityAssurance:l10n_Search_Verifier|wiki]]
| runnable, passing
| runnable, passing
| Info is current, but current plans are unclear.
| Info is current, but current plans are unclear.
|-
|-
| style="background:#00ff00" |
| style="background:#00ff00" |
| XSLT
| [http://lxr.mozilla.org/mozilla/source/content/xslt/tests/buster content/xslt/tests/buster]
| re-purposing of the Adobe Xalan tests.
| re-purposing of the Adobe Xalan tests.
| [[Custom Test Executables|custom executable/C++]]
| [[Custom Test Executables|custom executable/C++]]
Line 229: Line 237:
| style="background:#ffff00" |
| style="background:#ffff00" |
| XBL2
| XBL2
| [http://wiki.mozilla.org/MozillaQualityAssurance:Test_Development_Projects#Platform_Content_Area wiki]
| [[MozillaQualityAssurance:Test_Development_Projects#Platform_Content_Area|wiki]]
|  
|  
| XBL2 still in design
| XBL2 still in design
Line 258: Line 266:
| Java Applets
| Java Applets
| list suggested by Java team at Apple
| list suggested by Java team at Apple
| browser and eyeballs, [http://wiki.mozilla.org/MozillaQualityAssurance:Test_JavaApplets wiki]
| browser and eyeballs, [[MozillaQualityAssurance:Test_JavaApplets|wiki]]
| Just a list. No automation is currently in process.
| Just a list. No automation is currently in process.
| Info is current. See [mailto:rkiddy@mozilla.com ray].
| Info is current. See [mailto:rkiddy@mozilla.com ray].
Line 278: Line 286:
| style="background:#00ff00" |
| style="background:#00ff00" |
| HTML Parsing
| HTML Parsing
| Will need to deal with WHATWG/html5lib tests [https://bugzilla.mozilla.org/show_bug.cgi?id=366936 366936]. The htmlparser/tests/html test vs a baseline, but a baseline has not been stored.
| Will need to deal with WHATWG/html5lib tests {{bug|366936}}. The htmlparser/tests/html test vs a baseline, but a baseline has not been stored.
| [[Custom Test Executables|custom executable/C++]]  
| [[Custom Test Executables|custom executable/C++]]  
|
|
Line 305: Line 313:
|-
|-
| style="background:#ffff00" |
| style="background:#ffff00" |
| Math/ML
| MathML
| Do mathematical glyphs display correctly on all platforms?
| Do mathematical glyphs display correctly on all platforms?
| "browsers and eyeballs"
| "browsers and eyeballs"
Line 373: Line 381:
|  
|  
| Info is in source, but not sure of source for more info.
| Info is in source, but not sure of source for more info.
|}
= External Resources/Tests =
{| class="fullwidth-table"
!Test&nbsp;Resource
!Brief&nbsp;Description
!Status&nbsp;of&nbsp;the&nbsp;Instructure
!Notes
|-
| http://www.propagandism.org/tests/domts/
| Mentioned in Bug 346627.
| Runnable with FF 2.0.0.8 but shows errors. Does not seem to run on FF 3 (custom build).
|
|-
| http://test.bclary.com/
| Not sure. Anybody can say? Mentioned in Bug 346627.
|
|
|-
| http://www.w3.org/DOM/Test/
| W3. What else is there to say? Mentioned in Bug 346627.
|
|
|-
| http://fantasai.inkedblade.net/style/tests/css2.1/
|
|
|
|-
| http://www.w3.org/International/tests/#rtl-chrome
|
|
|
|-
| http://www.mozilla.org/projects/mathml/demo/tester.html
|
|
|
|-
| http://cgi.din.or.jp/~hagi3/JavaScript/Mozilla/SampleList.cgi?fmt=html
| List of Mozilla DOM Samples
|
|
|-
| http://mochikit.com/examples/effects/full.html
|
|
|
|-
| http://www.maths-informatique-jeux.com/international/mathml_test/index.xml
|
|
|
|-
| http://cgi.din.or.jp/~hagi3/JavaScript/Mozilla/SampleList.cgi?fmt=html
|
|
|
|-
| http://bclary.com/projects/spider/
|
|
|
|-
| http://www.webstandards.org/action/acid2/
| Acid2 Browser Test
|
|
|-
| http://hixie.ch/tests/
|
|
|
|-
| http://www.cynthiasays.com/
| Cynthia Says™ Portal
|
|
|-
| http://webxact.watchfire.com/
| watchfire WebXACT
|
|
|-
| http://www.tawdis.net/taw3/cms/en
| TAW (Web Accessibility Test)
|
|
|}
|}


Line 378: Line 475:


{| class="fullwidth-table"
{| class="fullwidth-table"
| style="background:#efefef" | '''Test Infrastructure Name'''
!TestInfrastructure Name
| style="background:#efefef" | '''Brief Description'''
!Brief Description
| style="background:#efefef" | '''Harnesses and Test Suites in the Infrastructure'''
!Harnesses and Test Suites in the Infrastructure
| style="background:#efefef" | '''Status of the Instructure'''
!Status of the Instructure
| style="background:#efefef" | '''Notes'''
!Notes
|-  
|-  
| tinderboxen
| tinderboxen
Line 393: Line 490:
| Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins.
| Continuous build engines, written in Python, cycling rapidly based on Bonsai checkins.
| "make check", reftest and MochiTest
| "make check", reftest and MochiTest
| reported via [http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest MozillaTest] tinderbox
| reported via [http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox Firefox] tinderbox
|-
|-
| QA Farm
| QA Farm
Jhammel
947

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/50679...218191"