Bugzilla:QA: Difference between revisions

Latest revision as of 13:08, 5 August 2016

QA team

The QA team was created on July 11, 2005, two days after the release of Bugzilla 2.18.3, to improve the quality of future releases. Bugzilla 2.20, which was released two months later, became our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team which found several tens of bugs. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team had less than 10 members) to test the most important features of Bugzilla.

As repeating the same tests manually again and again for each new release quickly became boring, we decided to automate the process as much as possible using Selenium. The QA team uses Perl Selenium scripts.

How to contribute?

As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the #bugzilla channel on IRC telling us that you are interested in contributing. Of course, you can also report bugs you discover to b.m.o directly. If you want to help with automated tests, you are highly encouraged to read below, and download our existing test scripts.

Running Selenium scripts

A read-only copy of the current Selenium scripts can be downloaded using git:

git clone https://github.com/bugzilla/bugzilla.git

A read/write copy can also be downloaded:

git clone git@github.com:bugzilla/bugzilla.git

A script named xt/config/generate_test_data.pl will automatically populate new installations, allowing the Selenium and WebService tests to be executed.

Note 1: The Bugzilla installation to test must already exist, and must already have its important parameters configured (i.e. urlbase, cookiepath, and mail_delivery_method). It is recommended that mail_delivery_method be set to Test, unless "actual" bug mail is desired.

Note 2: Before executing generate_test_data.pl, make sure the parameters in xt/config/selenium_test.conf are set correctly. This configuration file must match the Bugzilla configuration, especially the urlbase parameter and the path to the browser. If mail_delivery_method is set to Test, fake user accounts can be defined in the config file.

Once the DB is populated, start the Selenium server and execute the scripts in xt/selenium. To start the Selenium server, type:

java -jar /path/to/selenium-server-standalone-2.x.x.jar

The Selenium Server can be downloaded from the Selenium website. To execute the scripts, go into the t/ directory and enter:

prove -v --timer *.t

-v will make the output verbose and --timer will display the time it takes to execute each script. Both options are optional. Note that the Test::WWW::Selenium Perl module must be installed in order for the scripts to execute. It is the interface between Perl and Selenium!

Writing Selenium scripts

The fastest way to write new Selenium scripts is to use the Selenium IDE extension for Firefox (note that Selenium IDE 1.0.12 is the last version to export Selenium scripts to the Perl language! Starting with version 1.1.0, the Perl exporter is no longer available). It records all your actions and converts them into a valid Selenium script. You can also write Selenium scripts manually with a text editor, but this is longer and can be pretty painful.

Each command is of the form $sel->command_name_ok('field', 'value', 'description'), where $sel is the test object created by Test::WWW::Selenium. The description is optional, but may give some useful information. All available commands are listed here, and are fully documented.

There are many bugs which need a test. You can pick one of them and write a test, either by improving one of the existing tests, or by writing a completely new one. There are also some other existing scripts which need improvements. If you want to submit a script, please file it here.

Ja

@@ Line 2: / Line 2: @@
 The QA team was created on July 11, 2005, two days after the release of Bugzilla 2.18.3, to improve the quality of future releases. [http://www.bugzilla.org/releases/2.20/ Bugzilla 2.20], which was released two months later, became our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team which found several tens of bugs. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team had less than 10 members) to test the most important features of Bugzilla.
-As repeating the same tests manually again and again for each new release quickly became boring, we decided to automate the process as much as possible using [http://seleniumhq.org/ Selenium]. For Bugzilla 2.20.x, 2.22.x and 3.0.x, we used [http://landfill.bugzilla.org/selenium/bugzilla/index.html test installations on landfill], which are using HTML Selenium scripts. Since Bugzilla 3.2 RC1, the QA team uses [http://bzr.bugzilla.org/selenium/ Perl Selenium scripts], which offer many advantages over their HTML equivalent.
+As repeating the same tests manually again and again for each new release quickly became boring, we decided to automate the process as much as possible using [http://seleniumhq.org/ Selenium]. The QA team uses [https://git.mozilla.org/?p=bugzilla/bugzilla.git;a=tree;f=xt;hb=HEAD Perl Selenium scripts].
 == How to contribute? ==
-As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the [irc://irc.mozilla.org/qa-bugzilla #qa-bugzilla] channel on IRC, or to write to [mailto:qa@bugzilla.org qa@bugzilla.org] telling us that you are interested in contributing. Of course, you can also report bugs you discover to [https://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla b.m.o] directly. If you want to help with automated tests, you are highly encouraged to read below, and download our existing test scripts.
+As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the [irc://irc.mozilla.org/bugzilla #bugzilla] channel on IRC telling us that you are interested in contributing. Of course, you can also report bugs you discover to [https://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla b.m.o] directly. If you want to help with automated tests, you are highly encouraged to read below, and download our existing test scripts.
-Since mid-2006, the QA team uses a [http://landfill.bugzilla.org/bugzillaqa Testopia installation] (basically, Bugzilla with the Testopia extension) to track testing progress and to let us easily manage remaining tests to do. That's a great tool to help us work efficiently, avoiding testing what has already been (automatically) tested. We still run some tests manually, despite more and more of them are converted to Selenium scripts (and to be honest, the number of tests to run manually now is close to zero). If you don't know how to write Selenium scripts, you can still help us either by writing new testcases, or by running existing ones manually. If you prefer to write Selenium scripts, that's even better!
 == Running Selenium scripts ==
-Till Bugzilla 3.0.x, Selenium tests were all written in HTML, which were the executed from a web browser. It has been decided for Bugzilla 3.2 RC1 and newer to convert them to Perl, giving us more flexibility and control. We no longer write test scripts in HTML.
+A read-only copy of the current Selenium scripts can be downloaded using git:
+  git clone https://github.com/bugzilla/bugzilla.git
-All our current Selenium scripts can be downloaded using bzr. For instance, to download Perl scripts for Bugzilla 3.4, type:
-  bzr co bzr://bzr.bugzilla.org/selenium/3.4
-Other available branches are listed [http://bzr.bugzilla.org/selenium/ here].
-Users having write access to our bzr repository must use this command:
- bzr co bzr+ssh://your_login@bzr.bugzilla.org/var/www/html/bzr/selenium/3.4
-where your_login is your user account name on landfill.
-'''Note:''' Do not expect the HTML scripts to work as is. They were based on a given test installation on landfill having some given user accounts, products, components and parameters, and so they won't run on a fresh test installation. They are only available so that you can see what we did till Bugzilla 3.0.
+A read/write copy can also be downloaded:
+ git clone git@github.com:bugzilla/bugzilla.git
-Bugzilla 3.2 and newer have a script named <em>config/generate_test_data.pl</em> which populates your fresh new installation automatically so that you can run our Selenium and WebService tests yourself on your own installation.
+A script named <em>xt/config/generate_test_data.pl</em> will automatically populate new installations, allowing the Selenium and WebService tests to be executed.
-'''Note 1:''' your Bugzilla installation must already exists, and you must already have configured most important parameters (such as the urlbase, cookiepath, and mail_delivery_method* parameters (* set it to 'Test' ideally, unless you really want to get bugmail)).
+'''Note 1:''' The Bugzilla installation to test must already exist, and must already have its important parameters configured (i.e. ''urlbase'', ''cookiepath'', and ''mail_delivery_method''). It is recommended that ''mail_delivery_method'' be set to '''Test''', unless "actual" bug mail is desired.
-'''Note 2:''' Before running <em>generate_test_data.pl</em>, make sure that you set parameters in <em>config/selenium_test.conf</em> correctly. This configuration file must match your local configuration, especially the url to your installation (urlbase), as well as the path to your browser ([http://seleniumhq.org/download/ Selenium RC 1.0.1] works like a charm with Firefox 3.5). If you set mail_delivery_method to 'Test', you can define fake user accounts in the config file.
+'''Note 2:''' Before executing <em>generate_test_data.pl</em>, make sure the parameters in <em>xt/config/selenium_test.conf</em> are set correctly. This configuration file must match the Bugzilla configuration, especially the ''urlbase'' parameter and the path to the browser. If ''mail_delivery_method'' is set to '''Test''', fake user accounts can be defined in the config file.
-Once the DB is populated, you can start the Selenium server and run our scripts located in t/. To start the Selenium server, you can either run <em>config/selenium_server_start.t</em> (which requires the [http://search.cpan.org/~lukec/Alien-SeleniumRC/ Alien::SeleniumRC] Perl module to be installed) or you can type:
+Once the DB is populated, start the Selenium server and execute the scripts in xt/selenium. To start the Selenium server, type:
-  java -jar /path/to/selenium-server.jar
+  java -jar /path/to/selenium-server-standalone-2.x.x.jar
-which is exactly what Alien::SeleniumRC is doing. If you didn't install Alien::SeleniumRC, then you must extract <em>selenium-server.jar</em> from the [http://seleniumhq.org/download/ Selenium RC] ZIP file (generally, the JAR file provided by SeleniumRC is more recent than the one found in Alien::SeleniumRC). To run the scripts, go into the t/ directory and type:
+The Selenium Server can be downloaded from the [http://seleniumhq.org/download/ Selenium] website. To execute the scripts, go into the t/ directory and enter:
   prove -v --timer *.t
--v will make the output verbose and --timer will display the time it takes to run each script. Both options are optional. Note that you must have the [http://search.cpan.org/~lukec/Test-WWW-Selenium/ Test::WWW::Selenium] Perl module installed in order to have the scripts to run as it will be the interface between Perl and Selenium!
+-v will make the output verbose and --timer will display the time it takes to execute each script. Both options are optional. Note that the [http://search.cpan.org/~lukec/Test-WWW-Selenium/ Test::WWW::Selenium] Perl module must be installed in order for the scripts to execute.  It is the interface between Perl and Selenium!
 == Writing Selenium scripts ==
-The fastest way to write new Selenium scripts is to use the [http://seleniumhq.org/download/ Selenium IDE] extension for Firefox. It records all your actions and converts them into a valid Selenium script. You can also write Selenium scripts manually with a text editor, but this is longer and can be pretty painful.
+The fastest way to write new Selenium scripts is to use the [http://seleniumhq.org/download/ Selenium IDE] extension for Firefox (note that [http://release.seleniumhq.org/selenium-ide/1.0.12/selenium-ide-1.0.12.xpi Selenium IDE 1.0.12] is the last version to export Selenium scripts to the Perl language! Starting with version 1.1.0, the Perl exporter is no longer available). It records all your actions and converts them into a valid Selenium script. You can also write Selenium scripts manually with a text editor, but this is longer and can be pretty painful.
-Each command is of the form $sel->command_name_ok('field', 'value', 'description'), where $sel is the test object created by Test::WWW::Selenium. The description is optional, but may give some useful information. All available commands are listed [http://search.cpan.org/~lukec/Test-WWW-Selenium/lib/WWW/Selenium.pm here], and are fully documented.
+Each command is of the form $sel->command_name_ok('field', 'value', 'description'), where $sel is the test object created by Test::WWW::Selenium. The description is optional, but may give some useful information. All available commands are listed [http://search.cpan.org/~mattp/Test-WWW-Selenium/lib/WWW/Selenium.pm here], and are fully documented.
-Selenium scripts being under development/review can be found in open bugs depending on [http://landfill.bugzilla.org/bugzillaqa/show_bug.cgi?id=3065 bug 3065]. Those shouldn't be considered as "safe" till they are available via bzr. There are also many other bugs which need a test. You can [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=FIXED+testcase%3F pick one of them] and write a test, either by improving one of the existing tests, or by writing a completely new one. If you want to submit a script, please file it [http://landfill.bugzilla.org/bugzillaqa/enter_bug.cgi?product=QA%20tests&component=Selenium%20scripts here], which is the Bugzilla installation used by the QA team.
+There are [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=FIXED+testcase%3F many bugs] which need a test. You can pick one of them and write a test, either by improving one of the existing tests, or by writing a completely new one. There are also some other [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=prod%3ABugzilla+comp%3AQA existing scripts] which need improvements. If you want to submit a script, please file it [https://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla&component=QA%20Test%20Scripts here].