QA/Automation/Projects/Mozmill Automation

Overview

Lead:	Henrik Skupin
Co-workers:	Al Billings, Anthony Hughes
Dates:	No end date set
Status:	Creating and Enhancing Scripts
Specs:	Add-on test-run script
Repository Location:	Automated Test-run Scripts]
Tracking Bug:	bug 562639

Excerpt

Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We plan to have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those will have to be run on any platform and also against localized builds of Firefox.

Project details

The way to reach the state when everything is automated is long can can be divided into the following sub-projects:

• Setting up a machine which can be used for automated tests
• Creation of scripts which cover all automation aspects
• Running the automation for release builds

Machine for Automated Testing

A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.

This sub-project has been finished. All requirements can be read on bug 537840

Automation Execution

Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.

Automation Scripts

The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:

• Download script to fetch builds from the FTP server (bug 528064
• Script for Smoketests, BFT, and FFT Test-runs
• Script for Software Update Test-runs (bug 564539)
• Script for Add-ons Test-run (bug 562445)

Software Update Script

To run the software update tests automatically a test-run script is required which will take care of any aspect of the tests to be run. Once it has been called it should automatically execute the steps below:

• Install a build
• Run the partial or complete software update tests
• Reinstall the build
• Run the fallback software update tests
• Reinstall the build
• Run background software update tests
• Uninstall the build

A script with this feature set has already been created on bug 504653 and is able to execute all the steps except step 6. The reason is that we do not have any automated tests for background software updates yet.

Script for Add-on Test-runs

While you would also be able to run tests against extensions directly with Mozmill itself, you would have to download the extension first and specify it as argument to the Mozmill command line client. Those steps can be simplified and will result in a new test-run script for add-ons.

It will become a Python script which can be used to run tests against all or only a particular extension. Once it has been called it should automatically execute the steps below:

• Perform tests against each listed extension
  • Download the latest version from the specified URL
  • Start Firefox with the extension installed
  • Run all available tests
  • Clean-up the system