QA/Automation/Projects/Crowd Testing/Documentation

With the Mozmill Crowd extension Mozilla QA wants to spread out the testing efforts by using Mozmill to run functional tests against Firefox and other Gecko based applications. It should be easy enough for everyone to execute existing Mozmill tests, without having to go through all the prerequisite steps to setup a whole test environment.

The target audience for the extension are testers who want to run functional tests with Mozmill against specific parts of the application, i.e. localizers who want to check their own locale of Firefox. From our perspective manual testing will never be completely covered by a single person, but having Mozmill all the available tests can be run in a split of time. It will give as a much broader insight how Firefox behaves for other mostly untested localized builds.

Objective

• Empower users of Firefox to run all of our automated functional Mozmill tests
• Make the setup of the Mozmill test environment transparent to the user
• Collect test results across all supported locales and geographical locations

Product Components

The Storage Folder

The central piece of the extension is its storage folder. It is used to hold all necessary tools and applications, and the test results and acquired test data. Due to limitations with the virtual environment scripts the path of the storage folder cannot contain a blank. That's why it cannot be automatically stored in the users Firefox profile, but has to be chosen by the user himself.

The following sub-folders exists in the storage:

Sub-Folder	Description
mozmill-automation	Mozmill automation scripts which will be executed by the extension
mozmill-env	The Mozmill test environment which is necessary to run Mozmill tests
mozmill-tests	The most recent set of Mozmill tests
reports	Reports which have been created during the test-runs.
screenshots	Screenshots which have been automatically created by some tests

The Test Environment

Mozmill Crowd needs a pre-configured Mozmill environment to be able to execute the Mozmill tests. Check the Mozmill Automation - Environment project for more details.

User Interface

All features the extension provides are bundled in one single window, which can be used to execute test-runs, and to analyze test results. In the future the window will probably also contain an editor, which let you fix broken tests and offers automatic patch creation.

For preferences which rarely change a preferences dialog is used. It's reachable via the main window or the add-ons manager.

Main Window

The main window of the Mozmill Crowd extension is the central point for test execution. It will allow the tester to select the application to test and the test-run to be executed. Further it displays the test results after the test-run has been finished.

With the dropdown menu users can select any version of Firefox, starting with Firefox 3.5, to run tests against. The locale of the application doesn't matter because all of our tests can be run with any of our supported localized Firefox builds. Formerly chosen applications will be cached in the dropdown menu for faster access. For the test-runs a fixed number of scripts are available. Those mostly correlate with our automation scripts in the mozmill-automation repository.

Preferences Dialog

To edit preferences which are rarely used a preferences dialog is available. It can be opened through the main window or via the add-ons manager.

Following settings can be altered:

• The location of the mozmill-tests repository. For the moment only the remote repository can be selected.
• Defines if reports have to be sent to the given report server.
• Opt-in to run add-on tests for extensions which are not hosted on addons.mozilla.org

Back-end

Overlayed by the front-end, the back-end features will not be visible to the user. Those are the heart of the extension, and will do their job silently. It's already notable in the setup step of the environment. Anyone who formerly used Mozmill itself, will know how complicated it can be to follow the installation steps. Now a single click is enough to prepare the system to run automated tests.

The back-end can be divided into 3 areas:

1. Setup of the test environment
2. Preparation of the required repositories
3. Execution of Mozmill tests
4. Sending reports to a central data cluster

Setup of the Test Environment

The test environment is different for all major platforms. The largest environment exists for Windows, which even has to ship the Python interpreter. On Linux and OS X we only have to serve and create a virtual environment for the system-wide installed Python environment. On all platforms we ensure to not pollute the users system. All data can be completely removed.

The test environments have been built as zip archives, and will be automatically downloaded and extracted by the extension into a user-defined folder.

Preparation of the Repositories

Two repositories are necessary to execute test-runs. the most important one is our Mozmill automation repository which contains all of the automation script. Further the Mozmill tests repository will be cloned to allow access to all of our existent Mozmill tests.

Both local clones will be updated before a test-run gets executed.

Execution of Mozmill Tests

Once the user has selected the application and test-run, a click onto the "Start Test-run" button will start the tests. The appropriate automation script is executed and takes care of all necessary steps.

The tests are getting executed in a fresh profile and a separate Firefox process. This ensures that the data in the daily profile will not be modified or deleted by a test or a possible dataloss bug in Firefox.

Test results will be stored in a report file on disk and send to the remote data cluster for further inspection and analysis. All of them can be checked on our Mozmill-Crowd dashboard.