SoftwareTesting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
(15 intermediate revisions by 6 users not shown)
Line 10: Line 10:
=Projects=
=Projects=
==Testing in General==
==Testing in General==
[[User:Davel|Dave Liebreich]] is interested in creating running code to test the mozilla codebase. Not just end-to-end automated tests, but unit tests, module tests, test harnesses, test reporting infrastructure, and anything else that lowers the barrier to creating new code that gives someone feedback.
We need your help! There is a small but growing set of software tools for testing Firefox functionality. Please see [http://developer.mozilla.org/en/docs/Mozilla_automated_testing MDC:Mozilla_automated_testing] for details on what we have currently.


Dave is also interested in extending and improving the existing testing process.  The information received from community testing efforts can be excellent and invaluable; Dave is looking for ways in which existing volunteers can improve their testing skill and skilled testers can join the effort.
Dave needs help.  The codebase is huge.  The community is huge.  The task is huge.
* [[SoftwareTesting:Watir and Firefox]] - Writing the code necessary for [http://wtr.rubyforge.org/ Watir] to drive Firefox
* [[SoftwareTesting:BBST and the Mozilla Codebase]] - Using mozilla-based apps and libraries as coursework examples and exercises in [http://testingeducation.org/BBST/index.html A Course in Black Box Testing]
* [[SoftwareTesting:TDD and the Mozilla Codebase]] - Infrastructure, support, and examples of using TDD in mozilla projects
==Black-Box Testing==
==Black-Box Testing==
* [[SoftwareTesting:BlackBoxTesting]]
* [[SoftwareTesting:BlackBoxTesting]]
Line 36: Line 29:
** xforms tests written using jsUnit
** xforms tests written using jsUnit
* FireWatir
* FireWatir
* jUnit/FIT + javaxpcom
* jUnit/FIT + javaxpcom - See [http://www.sirgroane.net/2010/09/automated-javascript-unit-tests-with-xpcshell-and-hudson howto integrate xpcshell and hudson]
* pyUnit/pyFIT + pyxpcom
* pyUnit/pyFIT + pyxpcom
* webdav extension test harness (precursor to some calendar tests)
* webdav extension test harness (precursor to some calendar tests)
Line 44: Line 37:


==Wishlist for xpcshell test harness==
==Wishlist for xpcshell test harness==
* Unbuffered stdout, to get a useful log file even when the test hangs
* Ability to run just a single test (with output to stdout rather than the logfile)
** In particular, ability to run a single test in a debugger
** Maybe as <code>make check TEST=test_authentication.js</code>, or as <code>./testone.sh test_authentication.js</code>
* Ability to mark tests as XFAIL when they are known to fail at the moment
* Ability to mark tests as XFAIL when they are known to fail at the moment
* non-fatal do_check_* (as new functions) to be able to see all the test failures, not just the first one
* non-fatal do_check_* (as new functions) to be able to see all the test failures, not just the first one
* print log file on stdout for failed tests, especially useful when tinderbox runs the tests and one fails


=Tools=
=Tools=
Line 65: Line 53:


=People=
=People=
* [[User:Davel|Dave Liebreich]]
* [[User:Rcampbell|Rob Campbell]]
* [[User:Rcampbell|Rob Campbell]]
* [http://developer.mozilla.org/en/docs/User:ChrisCooper Chris Cooper]
* [[User:Bhearsum|Ben Hearsum]]
* [[User:Rhelmer|Rob Helmer]]
* [[User:Anodelman|Alice Nodelman]]
* [[User:Ctalbert|Clint Talbert]]


=Ideas to Collect=
=Ideas to Collect=
(stuff that needs to be organised)
(stuff that needs to be organised)
* [[SoftwareTesting:Tools:jsUnit]]
* [[SoftwareTesting:Tools:SimpleXulTestHarness]]
* example extension test using foxunit
** davel thinks xul/js-run-as-chrome is probably better
* jsunit in chrome proof-of-concept - [http://lxr.mozilla.org/mozilla/source/toolkit/mozapps/update/src/testnsUpdateService.js file] and Makefile.in in the same directory
* dbaron's [http://lxr.mozilla.org/mozilla/source/layout/tools/reftest/ layout reftest stuff]
** uses copy of domi code to handle command-line args in both firefox and seamonkey
* continuous integration/test issues
** {{bug|351968}} filed to track ENABLE_TEST issues
** build team has decided to avoid spending time getting new things to work with tinderbox
*** we need a tinderbox replacement, so this would be wasted time
** if a new test can be run in a manner very similar to an existing tinderbox feature, then it probably will be added
* aus2 tests using pyFit/fitnesse
** check-in pending from morgamic
[[Category:SoftwareTesting]]

Latest revision as of 23:16, 7 September 2010

In The Ongoing Revolution in Software Testing (pdf), Cem Kaner defines testing as

[...] a technical investigation of a product, done to expose quality-related information.

This page is a starting point to find, learn about, and track technical investigations of the Mozilla codebase.

Please note: this catalog is not limited to investigations done by members of the Mozilla qa team and community.

Projects

Testing in General

We need your help! There is a small but growing set of software tools for testing Firefox functionality. Please see MDC:Mozilla_automated_testing for details on what we have currently.

Black-Box Testing

Software Update Testing

Project Ideas

  • Defining a common invocation mechanism for tests, with example code
  • Defining a common reporting format for tests
    • perl has TAP, xUnit has textrunner (. vs F)
  • fixture-based frameworks for test development
    • hosting of Fit/Fitnesse to run tests against a browser instance
    • tinderbox-based tests

Areas/concepts to investigate

Wishlist for xpcshell test harness

  • Ability to mark tests as XFAIL when they are known to fail at the moment
  • non-fatal do_check_* (as new functions) to be able to see all the test failures, not just the first one

Tools

People

Ideas to Collect

(stuff that needs to be organised)

  • SoftwareTesting:Tools:jsUnit
  • SoftwareTesting:Tools:SimpleXulTestHarness
  • example extension test using foxunit
    • davel thinks xul/js-run-as-chrome is probably better
  • jsunit in chrome proof-of-concept - file and Makefile.in in the same directory
  • dbaron's layout reftest stuff
    • uses copy of domi code to handle command-line args in both firefox and seamonkey
  • continuous integration/test issues
    • bug 351968 filed to track ENABLE_TEST issues
    • build team has decided to avoid spending time getting new things to work with tinderbox
      • we need a tinderbox replacement, so this would be wasted time
    • if a new test can be run in a manner very similar to an existing tinderbox feature, then it probably will be added
  • aus2 tests using pyFit/fitnesse
    • check-in pending from morgamic
Retrieved from "https://wiki.allizom.org/index.php?title=SoftwareTesting&oldid=251570"