SoftwareTesting

In The Ongoing Revolution in Software Testing (pdf), Cem Kaner defines testing as

[...] a technical investigation of a product, done to expose quality-related information.

This page is a starting point to find, learn about, and track technical investigations of the Mozilla codebase.

Please note: this catalog is not limited to investigations done by members of the Mozilla qa team and community.

• SoftwareTesting:Scratchpad

Projects

Testing in General

We need your help! There is a small but growing set of software tools for testing Firefox functionality. Please see MDC:Mozilla_automated_testing for details on what we have currently.

Black-Box Testing

• SoftwareTesting:BlackBoxTesting

Software Update Testing

• SoftwareTesting:SoftwareUpdateTesting

Project Ideas

• Defining a common invocation mechanism for tests, with example code
• Defining a common reporting format for tests
  • perl has TAP, xUnit has textrunner (. vs F)
• fixture-based frameworks for test development
  • hosting of Fit/Fitnesse to run tests against a browser instance
  • tinderbox-based tests

Areas/concepts to investigate

• xulUnit
• jsUnit
  • xforms tests written using jsUnit
• FireWatir
• jUnit/FIT + javaxpcom
• pyUnit/pyFIT + pyxpcom
• webdav extension test harness (precursor to some calendar tests)
• jsDriver.pl
  • what can we use/re-use
• SoftwareTesting:2006-06-26 Brainstorm

Wishlist for xpcshell test harness

• Unbuffered stdout, to get a useful log file even when the test hangs
• Ability to run just a single test (with output to stdout rather than the logfile)
  • In particular, ability to run a single test in a debugger
  • Maybe as make check TEST=test_authentication.js, or as ./testone.sh test_authentication.js
• Ability to mark tests as XFAIL when they are known to fail at the moment
• non-fatal do_check_* (as new functions) to be able to see all the test failures, not just the first one
• print log file on stdout for failed tests, especially useful when tinderbox runs the tests and one fails

Tools

• SoftwareTesting:Catalog of Automated Tests - a list of pointers to existing automated tests in the mozilla codebase
• bug test-harness tracking bug for work on mozilla test harnesses
• SoftwareTesting:Tools:Simple xpcshell test harness - project tracking page
• SoftwareTesting:Tools:JsshDriverPython
• SoftwareTesting:Tools:TestRunnerPython
• Updates to Testing Module
• Example code for simulating a click in js
• JSAN Version of Perl's Test::Simple and related modules
• bug test-suites tracking bug for adding tests to test suites
• How to shut down the browser

People

• Rob Campbell
• Chris Cooper
• Ben Hearsum
• Rob Helmer
• Alice Nodelman
• Clint Talbert

Ideas to Collect

(stuff that needs to be organised)

• SoftwareTesting:Tools:jsUnit
• SoftwareTesting:Tools:SimpleXulTestHarness
• example extension test using foxunit
  • davel thinks xul/js-run-as-chrome is probably better
• jsunit in chrome proof-of-concept - file and Makefile.in in the same directory
• dbaron's layout reftest stuff
  • uses copy of domi code to handle command-line args in both firefox and seamonkey
• continuous integration/test issues
  • bug 351968 filed to track ENABLE_TEST issues
  • build team has decided to avoid spending time getting new things to work with tinderbox
    • we need a tinderbox replacement, so this would be wasted time
  • if a new test can be run in a manner very similar to an existing tinderbox feature, then it probably will be added
• aus2 tests using pyFit/fitnesse
  • check-in pending from morgamic