QA/Automation/Projects/Crowd Testing/Documentation
With the Mozmill Crowd extension Mozilla QA wants to spread out the testing efforts by using Mozmill to run functional tests against Firefox and other Gecko based applications. It should be easy enough for everyone to execute existing Mozmill tests, without having to go through all the prerequisite steps to setup a whole test environment.
The target audience for the extension are testers who want to run functional tests with Mozmill against specific parts of the application, i.e. localizers who want to check their own locale of Firefox. From our perspective manual testing will never be completely covered by a single person, but having Mozmill all the available tests can be run in a split of time. It will give as a much broader insight how Firefox behaves for other mostly untested localized builds.
Objective
- Empower users of Firefox to run all of our automated functional Mozmill tests
- Make the setup of the Mozmill test environment transparent to the user
- Collect test results across all supported locales and geographical locations
Product Components
The Storage Folder
The central piece of the extension is its storage folder. It is used to hold all necessary tools and applications, and the test results and acquired test data. Due to limitations with the virtual environment scripts the path of the storage folder cannot contain a blank. That's why it cannot be automatically stored in the users Firefox profile, but has to be chosen by the user himself.
The following sub-folders exists in the storage:
| Sub-Folder | Description |
| mozmill-automation | Mozmill automation scripts which will be executed by the extension |
| mozmill-env | The Mozmill test environment which is necessary to run Mozmill tests |
| mozmill-tests | The most recent set of Mozmill tests |
| reports | Reports which have been created during the test-runs. |
| screenshots | Screenshots which have been automatically created by some tests |
The Test Environment
Before Mozmill tests can be run on a machine, the test environment has to be installed first. It includes all necessary tools and modules which are necessary to run Mozmill and to execute our automation scripts. The environment is a part of the storage folder.
Due to any operating system brings its own set of pre-installed software, the test environment will be different for each one. See the tables below which tools and modules the environments contain.
For the installation of additional Python modules a virtual environment is used. That ensures that we do not change any configuration of the users system and stay local in our sandbox. All the modules will be installed during the setup of the test environment. Update checks will happen in the background each time the Mozmill Crowd dialog gets opened by the user.
Windows
The biggest chunk of the test environment on Windows is the Python interpreter itself. It's necessary because it is not a default application on Windows. Otherwise there is no dependency on other software.
| Tool / Module | Updatability |
| Python 2.6.6 | No (requires new test environment) |
| Python Mercurial 1.7.x | No (requires new test environment) |
| Setuptools | No (requires new test environment) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
OS X
The setup of the test environment on OS X only works with the Python package installed with the system. Any other package installed i.e. via Mac Ports does not work, because the Python headers are not available.
| Tool / Module | Updatability |
| VirtualEnv | Limited (installed with initial setup) |
| Python Mercurial 1.7.x | Limited (installed with initial setup) |
| Setuptools | Limited (installed with virtualenv) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
Linux
The setup of the test environment only works when the Python headers are installed, i.e. for Debian and Ubuntu it will be python-dev.
| Tool / Module | Updatability |
| VirtualEnv | Limited (installed with initial setup) |
| Python Mercurial 1.7.x | Limited (installed with initial setup) |
| Setuptools | Limited (installed with virtualenv) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
User Interface
All features the extension provides are bundled in one single window, which can be used to execute test-runs, and to analyze test results. In the future the window will probably also contain an editor, which let you fix broken tests and offers automatic patch creation.
For preferences which rarely change a preferences dialog is used. It's reachable via the main window or the add-ons manager.
Main Window
The main window of the Mozmill Crowd extension is the central point for test execution. It will allow the tester to select the application to test and the test-run to be executed. Further it displays the test results after the test-run has been finished.
With the dropdown menu users can select any version of Firefox, starting with Firefox 3.5, to run tests against. The locale of the application doesn't matter because all of our tests can be run with any of our supported localized Firefox builds. Formerly chosen applications will be cached in the dropdown menu for faster access. For the test-runs a fixed number of scripts are available. Those mostly correlate with our automation scripts in the mozmill-automation repository.
Preferences Dialog
To edit preferences which are rarely used a preferences dialog is available. It can be opened through the main window or via the add-ons manager.
Following settings can be altered:
- The location of the mozmill-tests repository. For the moment only the remote repository can be selected.
- Defines if reports have to be sent to the given report server.
- Opt-in to run add-on tests for extensions which are not hosted on addons.mozilla.org
Backend
Project Details
Preparing the System
To run Mozmill tests, the system has to be prepared to match a specific set of requirements. The steps which are necessary aren't the easiest ones and can cause trouble, especially when you are doing it for the first time.
That's why the extension has to create a virtual environment, which will contain all needed applications and tools. The following steps will be necessary:
- Retrieving the Python interpreter (or bundle it with the extension)
- Create a virtual Python environment
- Install the latest release of Mozmill
- Retrieve the automation scripts
Executing Mozmill Tests
Given by the selected dropdown entry on the toolbar the Python script of the related test-run gets executed. It will perform the following actions:
- Cloning the Mozmill tests from the mozmill-tests repository
- Updating the tests branch to the Gecko branch of the currently running version of Firefox
- Starting another instance of Firefox
- Execute all existent tests
Results Analysis
Once a test-run has been finished the results can be uploaded to our report server. There they can be examined and checked for new failures. A web dashboard will be created to offer the most valuable information.