QA/Automation/Projects/Crowd Testing/Documentation
With the Mozmill Crowd extension Mozilla QA wants to spread out the testing efforts by using Mozmill to run functional tests against Firefox and other Gecko based applications. It should be easy enough for everyone to execute existing Mozmill tests, without having to go through all the prerequisite steps to setup a whole test environment.
The target audience for the extension are testers who want to run functional tests with Mozmill against specific parts of the application, i.e. localizers who want to check their own locale of Firefox. From our perspective manual testing will never be completely covered by a single person, but having Mozmill all the available tests can be run in a split of time. It will give as a much broader insight how Firefox behaves for other mostly untested localized builds.
Objective
- Empower users of Firefox to run all of our automated functional Mozmill tests
- Make the setup of the Mozmill test environment transparent to the user
- Collect test results across all supported locales and geographical locations
Product Components
The Storage Folder
The central piece of the extension is its storage folder. It is used to hold all necessary tools and applications, and the test results and acquired test data. Due to limitations with the virtual environment scripts the path of the storage folder cannot contain a blank. That's why it cannot be automatically stored in the users Firefox profile, but has to be chosen by the user himself.
The following sub-folders exists in the storage:
| Sub-Folder | Description |
| mozmill-automation | Mozmill automation scripts which will be executed by the extension |
| mozmill-env | The Mozmill test environment which is necessary to run Mozmill tests |
| mozmill-tests | The most recent set of Mozmill tests |
| reports | Reports which have been created during the test-runs. |
| screenshots | Screenshots which have been automatically created by some tests |
The Test Environment
Before Mozmill tests can be run on a machine, the test environment has to be installed first. It includes all necessary tools and modules which are necessary to run Mozmill and to execute our automation scripts. The environment is a part of the storage folder.
Due to any operating system brings its own set of pre-installed software, the test environment will be different for each one. See the tables below which tools and modules the environments contain.
For the installation of additional Python modules a virtual environment is used. That ensures that we do not change any configuration of the users system and stay local in our sandbox. All the modules will be installed during the setup of the test environment. Update checks will happen in the background each time the Mozmill Crowd dialog gets opened by the user.
Windows
The biggest chunk of the test environment on Windows is the Python interpreter itself. It's necessary because it is not a default application on Windows. Otherwise there is no dependency on other software.
| Tool / Module | Updatability |
| Python 2.6.6 | No (requires new test environment) |
| Python Mercurial 1.7.x | No (requires new test environment) |
| Setuptools | No (requires new test environment) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
OS X
The setup of the test environment on OS X only works with the Python package installed with the system. Any other package installed i.e. via Mac Ports does not work, because the Python headers are not available.
| Tool / Module | Updatability |
| VirtualEnv | Limited (installed with initial setup) |
| Python Mercurial 1.7.x | Limited (installed with initial setup) |
| Setuptools | Limited (installed with virtualenv) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
Linux
The setup of the test environment only works when the Python headers are installed, i.e. for Debian and Ubuntu it will be python-dev.
| Tool / Module | Updatability |
| VirtualEnv | Limited (installed with initial setup) |
| Python Mercurial 1.7.x | Limited (installed with initial setup) |
| Setuptools | Limited (installed with virtualenv) |
| Mozmill | Yes (background check on start-up) |
| JSbridge | Yes (background check on start-up) |
| Mozrunner | Yes (background check on start-up) |
User Interface
All features the extension provides are bundled in one single window, which can be used to execute test-runs, and to analyze test results. In the future the window will probably also contain an editor, which let you fix broken tests and offers automatic patch creation.
For preferences which rarely change a preferences dialog is used. It's reachable via the main window or the add-ons manager.
Main Window
The main window of the Mozmill Crowd extension is the central point for test execution. It will allow the tester to select the application to test and the test-run to be executed. Further it displays the test results after the test-run has been finished.
With the dropdown menu users can select any version of Firefox, starting with Firefox 3.5, to run tests against. The locale of the application doesn't matter because all of our tests can be run with any of our supported localized Firefox builds. Formerly chosen applications will be cached in the dropdown menu for faster access. For the test-runs a fixed number of scripts are available. Those mostly correlate with our automation scripts in the mozmill-automation repository.
Preferences Dialog
To edit preferences which are rarely used a preferences dialog is available. It can be opened through the main window or via the add-ons manager.
Following settings can be altered:
- The location of the mozmill-tests repository. For the moment only the remote repository can be selected.
- Defines if reports have to be sent to the given report server.
- Opt-in to run add-on tests for extensions which are not hosted on addons.mozilla.org
Back-end
Overlayed by the front-end, the back-end features will not be visible to the user. Those are the heart of the extension, and will do their job silently. It's already notable in the setup step of the environment. Anyone who formerly used Mozmill itself, will know how complicated it can be to follow the installation steps. Now a single click is enough to prepare the system to run automated tests.
The back-end can be divided into 3 areas:
- Setup of the test environment
- Preparation of the required repositories
- Execution of Mozmill tests
- Sending reports to a central data cluster
Setup of the Test Environment
The test environment is different for all major platforms. The largest environment exists for Windows, which even has to ship the Python interpreter. On Linux and OS X we only have to serve and create a virtual environment for the system-wide installed Python environment. On all platforms we ensure to not pollute the users system. All data can be completely removed.
The test environments have been built as zip archives, and will be automatically downloaded and extracted by the extension into a user-defined folder.
Preparation of the Repositories
Two repositories are necessary to execute test-runs. the most important one is our Mozmill automation repository which contains all of the automation script. Further the Mozmill tests repository will be cloned to allow access to all of our existent Mozmill tests.
Both local clones will be updated before a test-run gets executed.
Execution of Mozmill Tests
Once the user has selected the application and test-run, a click onto the "Start Test-run" button will start the tests. The appropriate automation script is executed and takes care of all necessary steps.
The tests are getting executed in a fresh profile and a separate Firefox process. This ensures that the data in the daily profile will not be modified or deleted by a test or a possible dataloss bug in Firefox.
Test results will be stored in a report file on disk and send to the remote data cluster for further inspection and analysis. All of them can be checked on our Mozmill-Crowd dashboard.