Bugzilla:QA: Difference between revisions

← Older edit

Bugzilla:QA (view source)

Revision as of 13:08, 5 August 2016

1,299 bytes removed , 5 August 2016

Update to reference Github

Gerv

Account confirmers, Anti-spam team, Confirmed users, Bureaucrats and Sysops emeriti

4,925

edits

@@ Line 1: / Line 1: @@
 == QA team ==
-Since July 11, 2005, [http://www.bugzilla.org/news two days] after the release of Bugzilla 2.18.3, the QA team has been created to improve the quality of future releases. [http://www.bugzilla.org/releases/2.20/ Bugzilla 2.20] has been our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team has less than 10 members) to test the most important features of Bugzilla.
+The QA team was created on July 11, 2005, two days after the release of Bugzilla 2.18.3, to improve the quality of future releases. [http://www.bugzilla.org/releases/2.20/ Bugzilla 2.20], which was released two months later, became our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team which found several tens of bugs. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team had less than 10 members) to test the most important features of Bugzilla.
-As repeating the same tests manually again and again for each new release quickly became rather boring, we tried to automate the process as much as possible. To help us in this task, the QA team uses a [http://www.openqa.org/selenium/index.html Selenium] installation on [http://landfill.bugzilla.org/selenium/bugzilla/index.html landfill] (access restricted, sorry) which can be run from a web browser. That's how Bugzilla 2.20.x, 2.22.x and 3.0.x are being tested. Since Bugzilla 3.2 RC1, the QA team uses Selenium scripts written in Perl, which offers many advantages over their HTML equivalent.
+As repeating the same tests manually again and again for each new release quickly became boring, we decided to automate the process as much as possible using [http://seleniumhq.org/ Selenium]. The QA team uses [https://git.mozilla.org/?p=bugzilla/bugzilla.git;a=tree;f=xt;hb=HEAD Perl Selenium scripts].
 == How to contribute? ==
-As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the [irc://irc.mozilla.org/qa-bugzilla #qa-bugzilla] channel on IRC, or to write to [mailto:qa@bugzilla.org qa@bugzilla.org] telling us that you are interested. Of course, you can also report bugs you discovered to [https://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla b.m.o] directly.
+As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the [irc://irc.mozilla.org/bugzilla #bugzilla] channel on IRC telling us that you are interested in contributing. Of course, you can also report bugs you discover to [https://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla b.m.o] directly. If you want to help with automated tests, you are highly encouraged to read below, and download our existing test scripts.
-Since mid-2006, we use a [http://landfill.bugzilla.org/bugzillaqa Testopia installation] to track testing progress and to let us easily manage remaining tests to do. See e.g. [http://landfill.bugzilla.org/bugzillaqa/tr_list_runs.cgi?plan_id=4 tests we did for Bugzilla 3.0]. That's a great tool to help us work efficiently, avoiding testing what has already been (automatically) tested.
+== Running Selenium scripts ==
+A read-only copy of the current Selenium scripts can be downloaded using git:
+ git clone https://github.com/bugzilla/bugzilla.git
-We still do some tests manually, despite more and more of them are converted to Selenium scripts. If you don't know how to write Selenium scripts, you can still help us either by writing new testcases, or by running existing ones manually. If you prefer to write Selenium scripts, that's even better!
+A read/write copy can also be downloaded:
+ git clone git@github.com:bugzilla/bugzilla.git
-== Writing Selenium scripts ==
+A script named <em>xt/config/generate_test_data.pl</em> will automatically populate new installations, allowing the Selenium and WebService tests to be executed.
-Scripts executed from a web browser and those called from a Perl script use a different syntax, the first ones being pure HTML, the second ones being written using Perl language.
-=== Scripts running from a web browser ===
+'''Note 1:''' The Bugzilla installation to test must already exist, and must already have its important parameters configured (i.e. ''urlbase'', ''cookiepath'', and ''mail_delivery_method''). It is recommended that ''mail_delivery_method'' be set to '''Test''', unless "actual" bug mail is desired.
-Till Bugzilla 3.0.x, Selenium tests were all written in HTML. It has been decided for Bugzilla 3.2 RC1 and newer to convert them to Perl, giving us more flexibility and control. We no longer write test scripts in HTML.
-=== Perl Selenium scripts ===
+'''Note 2:''' Before executing <em>generate_test_data.pl</em>, make sure the parameters in <em>xt/config/selenium_test.conf</em> are set correctly. This configuration file must match the Bugzilla configuration, especially the ''urlbase'' parameter and the path to the browser. If ''mail_delivery_method'' is set to '''Test''', fake user accounts can be defined in the config file.
-The fastest way to write Selenium scripts is to use [http://openqa.org/selenium-ide/ selenium-IDE] ([https://addons.mozilla.org/extensions/moreinfo.php?application=firefox&category=Newest&numpg=10&id=2079 amo], formerly [http://seleniumrecorder.mozdev.org/ Selenium Recorder]). Selenium-IDE is a Firefox extension which writes scripts for you. It records your actions and converts them into a valid Selenium script. If you decide to install this extension, you don't need to install Selenium separately; everything is included in the XPI package (samples and docs are not included though). You can also write Selenium scripts with a text editor, but this is longer and can be pretty painful.
-In order to run these Perl scripts, you first have to install 2 Perl modules, both of which are available on CPAN:
+Once the DB is populated, start the Selenium server and execute the scripts in xt/selenium. To start the Selenium server, type:
-  [http://search.cpan.org/~lukec/Alien-SeleniumRC/ Alien::SeleniumRC]
+  java -jar /path/to/selenium-server-standalone-2.x.x.jar
- [http://search.cpan.org/~lukec/Test-WWW-Selenium/ Test::WWW::Selenium]
-Each command is then of the form $sel->command_name_ok('field', 'value', 'description'), where $sel is the test object created by Test::WWW::Selenium and which will do the interface between your test script and your Bugzilla installation. The description is optional, but gives very useful information. A typical output is of the form:
+The Selenium Server can be downloaded from the [http://seleniumhq.org/download/ Selenium] website. To execute the scripts, go into the t/ directory and enter:
+ prove -v --timer *.t
- #perl test_enter_new_bug.t
+-v will make the output verbose and --timer will display the time it takes to execute each script. Both options are optional. Note that the [http://search.cpan.org/~lukec/Test-WWW-Selenium/ Test::WWW::Selenium] Perl module must be installed in order for the scripts to execute.  It is the interface between Perl and Selenium!
- ok 1 - open, /bugzilla/
- ok 2 - Enter admin login name
- ok 3 - Enter admin password
- ok 4 - Submit credentials
- ok 5 - open, /bugzilla/enter_bug.cgi?product=TestProduct
- ok 6 - Display enter_bug.cgi for the selected product (bypass classifications)
- ok 7 - Enter bug summary
- ok 8 - Enter bug description
- ok 9 - Submit bug data to post_bug.cgi
-  ok 10 - Bug created
-..10
-What you can read here are the descriptions given for each command of the script. This makes debugging much easier!
+== Writing Selenium scripts ==
+The fastest way to write new Selenium scripts is to use the [http://seleniumhq.org/download/ Selenium IDE] extension for Firefox (note that [http://release.seleniumhq.org/selenium-ide/1.0.12/selenium-ide-1.0.12.xpi Selenium IDE 1.0.12] is the last version to export Selenium scripts to the Perl language! Starting with version 1.1.0, the Perl exporter is no longer available). It records all your actions and converts them into a valid Selenium script. You can also write Selenium scripts manually with a text editor, but this is longer and can be pretty painful.
-Existing Selenium scripts can be found in bugs depending on [http://landfill.bugzilla.org/bugzillaqa/show_bug.cgi?id=3065 bug 3065]. Members of the QA team can use CVS to get them all at once using:
- cvs -d my_login@landfill.bugzilla.org:/cvsroot co selenium
-where my_login is your user account name on landfill.
+Each command is of the form $sel->command_name_ok('field', 'value', 'description'), where $sel is the test object created by Test::WWW::Selenium. The description is optional, but may give some useful information. All available commands are listed [http://search.cpan.org/~mattp/Test-WWW-Selenium/lib/WWW/Selenium.pm here], and are fully documented.
-Once you have the repository on your local disk, all you have to do is to edit bugzilla32/config/selenium_test.conf to match your local configuration, especially the url to your installation, which must already exist (but should contain nothing more than what checksetup.pl created, i.e. your admin account, the TestProduct and system groups), as well as the path to your browser (if possible Firefox 2 due to a problem with Firefox 3, the EULA license being displayed on startup with a new profile, preventing Selenium to work). If you set mail_delivery_method to 'Testfile', you can define fake user accounts in the config file. When the config file matches your local configuration, the last step is to run generate_test_data.pl, which will populate your DB. This last step is required as all Selenium and WebService scripts assume the DB to have some given data in it. If a test goes wrong and breaks your DB, all you have to do is to recreate an empty DB, run checksetup.pl for the initialization, and then generate_test_data.pl again to repopulate the DB.
+There are [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=FIXED+testcase%3F many bugs] which need a test. You can pick one of them and write a test, either by improving one of the existing tests, or by writing a completely new one. There are also some other [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=prod%3ABugzilla+comp%3AQA existing scripts] which need improvements. If you want to submit a script, please file it [https://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla&component=QA%20Test%20Scripts here].