SoftwareTesting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (spelling/caps fixes)
 
(43 intermediate revisions by 7 users not shown)
Line 1: Line 1:
= Overview =
In [http://www.kaner.com/pdfs/TheOngoingRevolution.pdf The Ongoing Revolution in Software Testing (pdf)], [http://www.kaner.com Cem Kaner] defines testing as
* Help Wanted


[[User:Davel|Dave Liebreich]] is interested in creating running code to test the mozilla codebase.  Not just end-to-end automated tests, but unit tests, module tests, test harnesses, test reporting infrastructure, and anything else that lowers the barrier to creating new code that gives someone feedback.
<blockquote cite="http://www.kaner.com/pdfs/TheOngoingRevolution.pdf">[...] a technical investigation of a product, done to expose quality-related information.</blockquote>


Dave needs help.  The codebase is huge.  The community is huge.  The task is huge.
This page is a starting point to find, learn about, and track technical investigations of the Mozilla codebase.


This page is the first cut at collecting Dave's plans and investigations.
Please note: this catalog is not limited to investigations done by members of the Mozilla qa team and community.


= Guiding principles =
* [[SoftwareTesting:Scratchpad]]
* Testing is good
=Projects=
* Testing provides technical information about the product/system under test
==Testing in General==
* There are lots of different types of testing, each providing a different type of information and involving different trade-offs
We need your help! There is a small but growing set of software tools for testing Firefox functionality. Please see [http://developer.mozilla.org/en/docs/Mozilla_automated_testing MDC:Mozilla_automated_testing] for details on what we have currently.
* replacing a test that is useful to person A with a test that is useful to person B but not useful to person A is bad - just add the new test
* automated developer testing works only if the developers are writing the tests.
** if developer finds something that is difficult to test, then developer can make code more testable
** if code is more testable, then tests can be more powerful
* feedback from user community is unstructured and somewhat unpredictable, but incredibly useful. We should not extinguish this valuable channel of information.
* developer unit tests are typically not-very-powerful tests. The value comes from
** immediate and precise feedback - written close to the code
** there are typically lots of them
** they function as executable specifications for the code, and can replace some documentation and usage examples


= Projects =
==Black-Box Testing==
* Catalog all existing test suites and/or test cases
* [[SoftwareTesting:BlackBoxTesting]]
** special page for security tests
==Software Update Testing==
* investigate existing testing tools
* [[SoftwareTesting:SoftwareUpdateTesting]]
** Watir (possibly using JSSh)
 
** xulUnit, jsUnit, jsUnit (yes, there are 2), xUnit, FIT
=Project Ideas=
* create test infrastructure architecture
* Defining a common invocation mechanism for tests, with example code
** common invocation and reporting formats for all tests
* Defining a common reporting format for tests
*** or choose 1 of no more than 3, if 1-size-does-not-fit-all
** perl has TAP, xUnit has textrunner (. vs F)
** put docs on wiki, code in cvs
* fixture-based frameworks for test development
** follow list of principles for automated testing
** hosting of Fit/Fitnesse to run tests against a browser instance
*** data-driven (data separate from test code)
** tinderbox-based tests
*** Keyword driven
==Areas/concepts to investigate==
**** action keywords
* xulUnit
**** fixtures to implement action keywords
* jsUnit
*** focus on oracle (how do we know the test passed or failed)
** xforms tests written using jsUnit
*** do not rule out computer-assisted manual testing (no automated oracle)
* FireWatir
*** use model-based test scripting when possible
* jUnit/FIT + javaxpcom - See [http://www.sirgroane.net/2010/09/automated-javascript-unit-tests-with-xpcshell-and-hudson howto integrate xpcshell and hudson]
**** allows for high volume automated testing
* pyUnit/pyFIT + pyxpcom
**** can walk the state transitions and detect failures other than asserts or crashes
* webdav extension test harness (precursor to some calendar tests)
* Community outreach
* jsDriver.pl
** identify people in the mozilla community who are already test-infected
** what can we use/re-use
** identify test-infected people outside the mozilla community and ask them to help
* [[SoftwareTesting:2006-06-26 Brainstorm]]
** hit up grad students who can make this work part of their academic projects
 
==Wishlist for xpcshell test harness==
* Ability to mark tests as XFAIL when they are known to fail at the moment
* non-fatal do_check_* (as new functions) to be able to see all the test failures, not just the first one
 
=Tools=
* [[SoftwareTesting:Catalog of Automated Tests]] - a list of pointers to existing automated tests in the mozilla codebase
* {{bug|test-harness}} tracking bug for work on mozilla test harnesses
* [[SoftwareTesting:Tools:Simple xpcshell test harness]] - project tracking page
* [[SoftwareTesting:Tools:JsshDriverPython]]
* [[SoftwareTesting:Tools:TestRunnerPython]]
* [http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=&branchtype=match&dir=mozilla%2Ftesting&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=month&mindate=&maxdate=&cvsroot=%2Fcvsroot Updates to Testing Module]
* [http://developer.mozilla.org/samples/domref/dispatchEvent.html Example code] for simulating a click in js
* [http://www.openjsan.org/doc/t/th/theory/Test/Simple/0.21/ JSAN Version] of Perl's Test::Simple and related modules
* {{bug|test-suites}} tracking bug for adding tests to test suites
* [[SoftwareTesting:Tools:BrowserShutdown|How to shut down the browser]]
 
=People=
* [[User:Rcampbell|Rob Campbell]]
* [http://developer.mozilla.org/en/docs/User:ChrisCooper Chris Cooper]
* [[User:Bhearsum|Ben Hearsum]]
* [[User:Rhelmer|Rob Helmer]]
* [[User:Anodelman|Alice Nodelman]]
* [[User:Ctalbert|Clint Talbert]]
 
=Ideas to Collect=
(stuff that needs to be organised)
* [[SoftwareTesting:Tools:jsUnit]]
* [[SoftwareTesting:Tools:SimpleXulTestHarness]]
* example extension test using foxunit
** davel thinks xul/js-run-as-chrome is probably better
* jsunit in chrome proof-of-concept - [http://lxr.mozilla.org/mozilla/source/toolkit/mozapps/update/src/testnsUpdateService.js file] and Makefile.in in the same directory
* dbaron's [http://lxr.mozilla.org/mozilla/source/layout/tools/reftest/ layout reftest stuff]
** uses copy of domi code to handle command-line args in both firefox and seamonkey
* continuous integration/test issues
** {{bug|351968}} filed to track ENABLE_TEST issues
** build team has decided to avoid spending time getting new things to work with tinderbox
*** we need a tinderbox replacement, so this would be wasted time
** if a new test can be run in a manner very similar to an existing tinderbox feature, then it probably will be added
* aus2 tests using pyFit/fitnesse
** check-in pending from morgamic
[[Category:SoftwareTesting]]

Latest revision as of 23:16, 7 September 2010

In The Ongoing Revolution in Software Testing (pdf), Cem Kaner defines testing as

[...] a technical investigation of a product, done to expose quality-related information.

This page is a starting point to find, learn about, and track technical investigations of the Mozilla codebase.

Please note: this catalog is not limited to investigations done by members of the Mozilla qa team and community.

Projects

Testing in General

We need your help! There is a small but growing set of software tools for testing Firefox functionality. Please see MDC:Mozilla_automated_testing for details on what we have currently.

Black-Box Testing

Software Update Testing

Project Ideas

  • Defining a common invocation mechanism for tests, with example code
  • Defining a common reporting format for tests
    • perl has TAP, xUnit has textrunner (. vs F)
  • fixture-based frameworks for test development
    • hosting of Fit/Fitnesse to run tests against a browser instance
    • tinderbox-based tests

Areas/concepts to investigate

Wishlist for xpcshell test harness

  • Ability to mark tests as XFAIL when they are known to fail at the moment
  • non-fatal do_check_* (as new functions) to be able to see all the test failures, not just the first one

Tools

People

Ideas to Collect

(stuff that needs to be organised)

  • SoftwareTesting:Tools:jsUnit
  • SoftwareTesting:Tools:SimpleXulTestHarness
  • example extension test using foxunit
    • davel thinks xul/js-run-as-chrome is probably better
  • jsunit in chrome proof-of-concept - file and Makefile.in in the same directory
  • dbaron's layout reftest stuff
    • uses copy of domi code to handle command-line args in both firefox and seamonkey
  • continuous integration/test issues
    • bug 351968 filed to track ENABLE_TEST issues
    • build team has decided to avoid spending time getting new things to work with tinderbox
      • we need a tinderbox replacement, so this would be wasted time
    • if a new test can be run in a manner very similar to an existing tinderbox feature, then it probably will be added
  • aus2 tests using pyFit/fitnesse
    • check-in pending from morgamic
Retrieved from "https://wiki.allizom.org/index.php?title=SoftwareTesting&oldid=251570"