QA/Mozmill Test Automation/Test Writing

Overview

Lead:	Henrik Skupin
Co-workers:	Anthony Hughes, Al Billings
Dates:	Finishing BFT tests end of Q1/2011
Status:	Implementation of BFT tests
Repository Location:	http://hg.mozilla.org/qa/mozmill-tests/
Tracking Document:	Mozmill Spreadsheet

Excerpt

Mozmill tests allow us to have a much broader coverage of manual tests as we ever could have when running tests by hand. The speed-up is incredible and allows us to run a dozen or even more test-runs in the same time as you would be able to do by hand. Therefore we are building up a complex scenario of tests from all possible areas, which will be run against all available builds of Firefox, on each platform, and in any locale.

This project covers the work which is needed to have such a complex test-suite in-place.

Project Details

In general there are about 700 manual tests on Litmus which could be automated. Those tests are separated into three different groups: Smoketests, BFT (Basic Functional Tests), and FFT (Full Functional Tests). While seeing that amount of tests are focus is currently on Smoketests and BFT tests. The reason therefore is that those tests have to be run for every security release of Firefox and need attention from QA. Having those automated would give us the chance to set the focus of manual testing to areas where check-ins happened for the new release. All other tests could be run automatically.

Once we have nearly finished the BFT testgroup, we will have to check if automating all FFT's is useful or not.

Identifying a Test Candidate

Once you have learned how to run Mozmill tests, you will probably have interests in creating a test yourself. The first step would be to check which tests already have been automated, which are in the works and which are remaining.

To track the ongoing work we have created a Google spreadsheet. The different tables show the status of the above mentioned testgroups. Checkout the BFT table to scan available subgroups which have to be automated. If you are interested in a subgroup check if there is already an owner set. Get in contact with him or send an email to Henrik, who can coordinate the work. We can figure out the necessary requirements which have to be fulfilled before work can start on this subgroup. If nothing has been done so far it's highly recommended to create a Shared Module as the first step before any test can be implemented.

Filing a Bug

After a test has been assigned to you, a bug has to be filed on Bugzilla. We have prepared a template which makes the creation as easy as possible.

Just obey the following steps:

• Add the description of the Litmus test to the summary
• Enter references to Litmus tests for all branches (3.7, 3.6, and 3.5) into the comment field

That's it. Now you can start...

Implementing a Test

At first you should read through the test writing guidelines to get familiar with the test creation.

Once it is clear check the available templates which have been prepared for a new test. Copy and paste the content into a new file to get the raw structure for your test. Finally fill in your details in the license plate.

Writing the Mozmill test means you should follow the steps given in the Litmus test as best as possible. If you have variations feel free to implement all of those. As more ways we are able to test as higher are the chances to find a regression in the future. Use available Shared Modules to create a test which is easy to understand by everyone.

To get the new test checked-in into our mozmill-test repository you will have to create a patch and go through the usual review cycle.

Verifying the Test works

Once the patch has been checked-in, the bug will be marked as fixed. We will wait for the next test-run on our machine in the QA lab. If no problems or regressions appear we are fine and you did a great job. Otherwise we have to back-out the patch and go back to the investigation phase and check why the test fails.