QA/Automation/Projects/Crowd Testing/Documentation
With the Mozmill Crowd extension Mozilla QA wants to spread out the testing efforts by using Mozmill to run functional tests against Firefox and other Gecko based applications. It should be easy enough for everyone to execute existing Mozmill tests, without having to go through all the prerequisite steps to setup a whole test environment.
The target audience for the extension are testers who want to run functional tests with Mozmill against specific parts of the application, i.e. localizers who want to check their own locale of Firefox. From our perspective manual testing will never be completely covered by a single person, but having Mozmill all the available tests can be run in a split of time. It will give as a much broader insight how Firefox behaves for other mostly untested localized builds.
Objective
- Empower users of Firefox to run all of our automated functional Mozmill tests
- Make the setup of the necessary test environment as easy as possible
- Collect test results across all supported locales and geographical locations
Product Components
TBD
Project Details
Only running Mozmill tests on buildbot or on the release-test machine in the QA lab is not sufficient to find misbehavior's of Firefox. A more wide-spread test-run is necessary. Such a possibility will be the result of the crowd testing project. An extension has to be developed which will automatically prepare the system, executes the test-run, and send the results to our report server for analysis.
The Extension
The easiest way to add additional features to Firefox is to install an extension. The same way will be used for this project. The extension, which will be offered on addons.mozilla.org, will add it's own toolbar right below of the existing ones. With a dropdown field the user will be able to select the wanted test-run, e.g. normal tests, update tests, or l10n tests. By clicking the 'Run' button the test-run gets executed.
Preparing the System
To run Mozmill tests, the system has to be prepared to match a specific set of requirements. The steps which are necessary aren't the easiest ones and can cause trouble, especially when you are doing it for the first time.
That's why the extension has to create a virtual environment, which will contain all needed applications and tools. The following steps will be necessary:
- Retrieving the Python interpreter (or bundle it with the extension)
- Create a virtual Python environment
- Install the latest release of Mozmill
- Retrieve the automation scripts
Executing Mozmill Tests
Given by the selected dropdown entry on the toolbar the Python script of the related test-run gets executed. It will perform the following actions:
- Cloning the Mozmill tests from the mozmill-tests repository
- Updating the tests branch to the Gecko branch of the currently running version of Firefox
- Starting another instance of Firefox
- Execute all existent tests
Results Analysis
Once a test-run has been finished the results can be uploaded to our report server. There they can be examined and checked for new failures. A web dashboard will be created to offer the most valuable information.