QA/Automation/Projects/Mozmill Automation

Overview

Lead:	Henrik Skupin
Co-workers:	Al Billings, Anthony Hughes
Dates:	No end date set
Status:	Creating and Enhancing Scripts
Specs:	Add-on test-run script
Repository Location:	Automated Test-run Scripts]
Tracking Bug:	bug 562639

Excerpt

Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We plan to have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those will have to be run on any platform and also against localized builds of Firefox.

Project details

The way to reach the state when everything is automated is long can can be divided into the following sub-projects:

• Setting up a machine which can be used for automated tests
• Creation of scripts which cover all automation aspects
• Running the automation for release builds

Machine for Automated Testing

A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.

This sub-project has been finished. All requirements can be read on bug 537840

Automation Execution

Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.

Automation Scripts

The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:

• Download script to fetch builds from the FTP server (bug 528064
• Script for Smoketests, BFT, and FFT tests
• Script for Software Update Test-runs
• Script for Add-ons Test-run (bug 562445)

Script for Add-on Test-runs

While you would also be able to run tests against extensions directly with Mozmill itself, you would have to download the extension first and specify it as argument to the Mozmill command line client. Those steps can be simplified and will result in a new test-run script for add-ons.

It will become a Python script which can be used to run tests against all or only a particular extension. Once it has been called it should automatically execute the steps below:

• Perform tests against each listed extension
  • Download the latest version from the specified URL
  • Start Firefox with the extension installed
  • Run all available tests
  • Clean-up the system