Bugzilla:QA

QA team

Since July 11, 2005, two days after the release of Bugzilla 2.18.3, the QA team has been created to improve the quality of future releases. Bugzilla 2.20 has been our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team has less than 10 members) to test the most important features of Bugzilla.

As repeating the same tests manually again and again for each new release quickly became rather boring, we tried to automate the process as much as possible. To help us in this task, the QA team uses a Selenium installation on landfill (access restricted, sorry) which can be run from a web browser. That's how Bugzilla 2.20.x, 2.22.x and 3.0.x are being tested. Since Bugzilla 3.2 RC1, the QA team uses Selenium scripts written in Perl, which offers many advantages over their HTML equivalent.

How to contribute?

As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the #qa-bugzilla channel on IRC, or to write to qa@bugzilla.org telling us that you are interested. Of course, you can also report bugs you discovered to b.m.o directly.

Since mid-2006, we use a Testopia installation to track testing progress and to let us easily manage remaining tests to do. See e.g. tests we did for Bugzilla 3.0. That's a great tool to help us work efficiently, avoiding testing what has already been (automatically) tested.

We still do some tests manually, despite more and more of them are converted to Selenium scripts. If you don't know how to write Selenium scripts, you can still help us either by writing new testcases, or by running existing ones manually. If you prefer to write Selenium scripts, that's even better!

Writing Selenium scripts

Scripts executed from a web browser and those called from a Perl script use a different syntax, the first ones being pure HTML, the second ones being written using Perl language.

Scripts running from a web browser

Till Bugzilla 3.0.x, Selenium tests were all written in HTML. It has been decided for Bugzilla 3.2 RC1 and newer to convert them to Perl, giving us more flexibility and control. We no longer write test scripts in HTML.

Perl Selenium scripts

The fastest way to write Selenium scripts is to use selenium-IDE (amo, formerly Selenium Recorder). Selenium-IDE is a Firefox extension which writes scripts for you. It records your actions and converts them into a valid Selenium script. If you decide to install this extension, you don't need to install Selenium separately; everything is included in the XPI package (samples and docs are not included though). You can also write Selenium scripts with a text editor, but this is longer and can be pretty painful.

In order to run these Perl scripts, you first have to install 2 Perl modules, both of which are available on CPAN:

Alien::SeleniumRC
Test::WWW::Selenium

Each command is then of the form $sel->command_name_ok('field', 'value', 'description'), where $sel is the test object created by Test::WWW::Selenium and which will do the interface between your test script and your Bugzilla installation. The description is optional, but gives very useful information. A typical output is of the form:

#perl test_enter_new_bug.t
ok 1 - open, /bugzilla/
ok 2 - Enter admin login name
ok 3 - Enter admin password
ok 4 - Submit credentials
ok 5 - open, /bugzilla/enter_bug.cgi?product=TestProduct
ok 6 - Display enter_bug.cgi for the selected product (bypass classifications)
ok 7 - Enter bug summary
ok 8 - Enter bug description
ok 9 - Submit bug data to post_bug.cgi
ok 10 - Bug created
1..10

What you can read here are the descriptions given for each command of the script. This makes debugging much easier!

Existing Selenium scripts can be found in bugs depending on bug 3065. Members of the QA team can use CVS to get them all at once using:

cvs -d my_login@landfill.bugzilla.org:/cvsroot co selenium

where my_login is your user account name on landfill.

Ja