QA/Automation/Projects/Crowd Testing/Documentation
With the Mozmill Crowd extension Mozilla QA wants to spread out the testing efforts by using Mozmill to run functional tests against Firefox and other Gecko based applications. It should be easy enough for everyone to execute existing Mozmill tests, without having to go through all the prerequisite steps to setup a whole test environment.
The target audience for the extension are testers who want to run functional tests with Mozmill against specific parts of the application, i.e. localizers who want to check their own locale of Firefox. From our perspective manual testing will never be completely covered by a single person, but having Mozmill all the available tests can be run in a split of time. It will give as a much broader insight how Firefox behaves for other mostly untested localized builds.
Objective
- Empower users of Firefox to run all of our automated functional Mozmill tests
- Make the setup of the Mozmill test environment transparent to the user
- Collect test results across all supported locales and geographical locations
Product Components
The Storage Folder
TBD
The Test Environment
Before Mozmill tests can be run on a machine, the test environment has to be installed first. It includes all necessary tools and modules which are necessary to run Mozmill and to execute our automation scripts. The environment is a part of the storage folder.
Due to any operating system brings its own set of pre-installed software, the test environment will be different for each one. See the tables below which tools and modules the environments contain.
For the installation of additional Python modules a virtual environment is used. That ensures that we do not change any configuration of the users system and stay local in our sandbox. All the modules will be installed during the setup of the test environment. Update checks will happen in the background each time the Mozmill Crowd dialog gets opened by the user.
Windows
The biggest chunk of the test environment on Windows is the Python interpreter itself. It's necessary because it is not a default application on Windows. Otherwise there is no dependency on other software.
| Tool / Module | Updatability |
| Python 2.6.6 | No (requires new test environment) |
| Python Mercurial 1.7.x | No (requires new test environment) |
| Setuptools | No (requires new test environment) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
OS X
The setup of the test environment on OS X only works with the Python package installed with the system. Any other package installed i.e. via Mac Ports does not work, because the Python headers are not available.
| Tool / Module | Updatability |
| VirtualEnv | Limited (installed with initial setup) |
| Python Mercurial 1.7.x | Limited (installed with initial setup) |
| Setuptools | Limited (installed with virtualenv) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
Linux
| Tool / Module | Updatability |
| VirtualEnv | Limited (installed with initial setup) |
| Python Mercurial 1.7.x | Limited (installed with initial setup) |
| Setuptools | Limited (installed with virtualenv) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
TBD
Project Details
The Extension
The easiest way to add additional features to Firefox is to install an extension. The same way will be used for this project. The extension, which will be offered on addons.mozilla.org, will add it's own toolbar right below of the existing ones. With a dropdown field the user will be able to select the wanted test-run, e.g. normal tests, update tests, or l10n tests. By clicking the 'Run' button the test-run gets executed.
Preparing the System
To run Mozmill tests, the system has to be prepared to match a specific set of requirements. The steps which are necessary aren't the easiest ones and can cause trouble, especially when you are doing it for the first time.
That's why the extension has to create a virtual environment, which will contain all needed applications and tools. The following steps will be necessary:
- Retrieving the Python interpreter (or bundle it with the extension)
- Create a virtual Python environment
- Install the latest release of Mozmill
- Retrieve the automation scripts
Executing Mozmill Tests
Given by the selected dropdown entry on the toolbar the Python script of the related test-run gets executed. It will perform the following actions:
- Cloning the Mozmill tests from the mozmill-tests repository
- Updating the tests branch to the Gecko branch of the currently running version of Firefox
- Starting another instance of Firefox
- Execute all existent tests
Results Analysis
Once a test-run has been finished the results can be uploaded to our report server. There they can be examined and checked for new failures. A web dashboard will be created to offer the most valuable information.