Bugzilla:QA
QA team
Since July 11, 2005, two days after the release of Bugzilla 2.18.3, the QA team has been created to improve the quality of future releases. Bugzilla 2.20 has been our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team has less than 10 members) to test the most important features of Bugzilla.
As repeating the same tests manually again and again for each new release quickly became rather boring, we tried to automate the process as much as possible. To help us in this task, the QA team uses a Selenium installation on landfill (access restricted, sorry) which can be run from a web browser. That's how Bugzilla 2.20.x, 2.22.x and 3.0.x are being tested. Since Bugzilla 3.2 RC1, the QA team uses Selenium scripts written in Perl, which offers many advantages over their HTML equivalent.
How to contribute?
As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the #qa-bugzilla channel on IRC, or to write to qa@bugzilla.org telling us that you are interested. Of course, you can also report bugs you discovered to b.m.o directly.
Since mid-2006, we use a Testopia installation to track testing progress and to let us easily manage remaining tests to do. See e.g. tests we did for Bugzilla 3.0. That's a great tool to help us work efficiently, avoiding testing what has already been (automatically) tested.
We still do some tests manually, despite more and more of them are converted to Selenium scripts. If you don't know how to write Selenium scripts, you can still help us either by writing new testcases, or by running existing ones manually. If you prefer to write Selenium scripts, that's even better!
Writing Selenium scripts
Scripts executed from a web browser and those called from a Perl script use a different syntax, the first ones being pure HTML, the second ones being written using Perl language.
Scripts running from a web browser
This is the easiest way to use Selenium. After having downloaded and untar'ed the tarball (we still use version 0.6.0 despite 0.8.1 is out, for compatibility issues), scripts simply consist of a table with three columns (command, field, value). A template is available in bug 317695 (template.html), as well as a useful Javascript file for most common actions (user-extensions.js).
selenium-IDE (amo, formerly Selenium Recorder) is a Firefox extension which writes scripts for you. It records your actions and converts them into a valid Selenium script. If you decide to install this extension, you don't need to install Selenium separately; everything is included in the XPI package (samples and docs are not included though).
Scripts called by a Perl program
The installation and configuration are a bit harder. First download and install Test::WWW::Selenium. In case you still use version 0.03, the following modules are required too:
requires:
Alien::Selenium Frontier::Client Frontier::RPC2 LWP v5.8 POE v0.29 POE::Component::Client::UserAgent POE::Component::Server::HTTP Test::LongString
build_requires:
HTTP::Daemon Test::More v0.47
When all these modules are installed, you will also need the SeleniumTest.pm module available in bug 317695 which is a simplified version of the one given with the Test-WWW-Selenium tarball. An example of a test script - 012qa.t - is also provided, which can e.g. be executed with runtests.pl, a Perl script available in the Bugzilla distribution.
For newer versions of Test::WWW::Selenium, requirements are different and SeleniumTest.pm is no longer required. Note that your old scripts running on 0.03 won't run on newer versions anymore.
As you can see, the syntax is different between HTML and Perl scripts, but the commands remain mostly the same. Make sure to end all your commands by "_ok" to generate an output on your screen. The format is now $test->command_name_ok('field', 'value', 'description'), where $test is the test object created by SeleniumTest.pm and which will do the interface between your test script and the web installation.
The description is optional, but gives very useful information. A typical output is of the form:
t/012qa....1..7 ok 1 - Load query.cgi ok 2 - Check title ok 3 - Enter and check input in the short_desc field ok 4 - Recheck the input (using a different method) ok 5 - Submit request (-> buglist.cgi) ok 6 - Check title ok 7 - Make sure that no bug is returned (0 bug found) ok All tests successful.
What you can read here are the descriptions given for each command of the script. This makes debugging much easier!