Bugzilla:QA
QA team
Since July 11, 2005, two days after the release of Bugzilla 2.18.3, the QA team has been created to improve the quality of future releases. Bugzilla 2.20 has been our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team has 9 members) to test the most important features of Bugzilla.
As repeating the same tests manually again and again for each new release quickly became rather boring, we tried to automate the process as much as possible. To help us in this task, the QA team use a Selenium installation on landfill which can either be run from a web browser or from a Perl script directly.
How to contribute?
As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the #qa-bugzilla channel on IRC, or to write to qa@bugzilla.org telling us that you are interested. Of course, you can also report bugs you discovered to b.m.o directly.
Since mid-2006, we use a Testopia installation to track testing progress and to let us easily manage remaining tests to do. See e.g. tests we did on October 2006 when we released 2.18.6, 2.20.3, 2.22.1 and 2.23.3. That's another way to help us efficiently, avoiding testing what has already been tested.
As we are still doing some tests manually, you can either do manual tests or choose one of the two approaches described below to write automated tests. In the latter case, and if you don't want to install all the required modules or cannot get them working, writing HTML scripts is probably what you want to do. One day, we will probably write a conversion tool, allowing us to write scripts in HTML and convert them in Perl and vice versa.
Writing Selenium scripts
Scripts executed from a web browser and those called from a Perl script use a different syntax, the first ones being pure HTML, the second ones being written using Perl language.
Scripts running from a web browser
This is the easiest way to use Selenium. After having downloaded and untar'ed the tarball (we still use version 0.6.0 despite 0.8.1 is out, for compatibility issues), scripts simply consist of a table with three columns (command, field, value). A template is available in bug 317695 (template.html), as well as a useful Javascript file for most common actions (user-extensions.js).
selenium-IDE (amo, formerly Selenium Recorder) is a Firefox extension which writes scripts for you. It records your actions and converts them into a valid Selenium script. If you decide to install this extension, you don't need to install Selenium separately; everything is included in the XPI package (samples and docs are not included though).
Scripts called by a Perl program
The installation and configuration are a bit harder. First download and install Test::WWW::Selenium. In case you still use version 0.03, the following modules are required too:
requires:
Alien::Selenium Frontier::Client Frontier::RPC2 LWP v5.8 POE v0.29 POE::Component::Client::UserAgent POE::Component::Server::HTTP Test::LongString
build_requires:
HTTP::Daemon Test::More v0.47
When all these modules are installed, you will also need the SeleniumTest.pm module available in bug 317695 which is a simplified version of the one given with the Test-WWW-Selenium tarball. An example of a test script - 012qa.t - is also provided, which can e.g. be executed with runtests.pl, a Perl script available in the Bugzilla distribution.
For newer versions of Test::WWW::Selenium, requirements are different and SeleniumTest.pm is no longer required. Note that your old scripts running on 0.03 won't run on newer versions anymore.
As you can see, the syntax is different between HTML and Perl scripts, but the commands remain mostly the same. Make sure to end all your commands by "_ok" to generate an output on your screen. The format is now $test->command_name_ok('field', 'value', 'description'), where $test is the test object created by SeleniumTest.pm and which will do the interface between your test script and the web installation.
The description is optional, but gives very useful information. A typical output is of the form:
t/012qa....1..7 ok 1 - Load query.cgi ok 2 - Check title ok 3 - Enter and check input in the short_desc field ok 4 - Recheck the input (using a different method) ok 5 - Submit request (-> buglist.cgi) ok 6 - Check title ok 7 - Make sure that no bug is returned (0 bug found) ok All tests successful.
What you can read here are the descriptions given for each command of the script. This makes debugging much easier!