QA/Automation/Projects/Mozmill Automation/Mozmill2Scripts

Overview

Name:	MozMill 2 Script Updates
Leads:	Dave Hunt, Henrik Skupin
Contributors:	David Guo
Repository:	https://github.com/whimboo/mozmill-automation
Bugzilla:	https://bugzilla.mozilla.org/show_bug.cgi?id=732134

Requirements

• Remove Mozmill wrapper dependency
• Remove Mozmill CLI dependency
• All past tests must be able to be run on both Mozmill 2.0 and 1.5
• Must be compatible with Thunderbird and Firefox
• Must be maintainable

Project Milestones

Milestone	Status	Description
Remove dependency of Mozmill wrapper	Complete; Implementation under review	Strategy: https://etherpad.mozilla.org/mozmill-wrapper-removal, Patch: https://bugzilla.mozilla.org/show_bug.cgi?id=732134
Determine if current automation scripts can adapt to Mozmill 2.0 infrastructure	Complete	The conclusion was that there are no viable options without rewriting some parts of TestRun; Detailed information: https://etherpad.mozilla.org/mozmill2strategydraft
Draft a spec and design the TestRun module remodel	Complete; Needs review	This effort will outline how TestRun will look inheriting from Mozrunner opposed to MozmillCLI
Implement the new TestRun	Not started	This effort will swap the Mozmill CLI inheritance with Mozrunner
Refactor and testing	Not started	After implementation is complete, the new TestRun module will be tested against previous tests and possible ways to refactor will be looked at

Architecture

Class Diagram

Data Flow

Components

To allow a relative flexible module usage and to make code re-usable for other projects most of the current code should be moved to mozbase or made available as packages on PyPI.

Module	Bug	Description
application	bug 769606	Application specific utilities. While updates will remain in our repository, the handling of the application.ini file could be part of mozrunner.
errors	n/a	Module which contains all the supported error classes. This module will remain in our repository.
json_file	n/a	Utilities for JSON files. This module will remain in our repository but should probably be moved to a generic file handling module.
install	bug 757411 (done)	Installer and uninstaller for Gecko based applications across platforms. This module should be merged with mozinstall.
rdf_parser	n/a	Parsing RDF files, e.g. install.rdf, with correct namespace handling. This module should be merged with mozprofile.
reports	Github (done)	Module for custom reports. It will remain in our repository and will work as plugin for Mozmill 2.0
repository	done	Utilities for handling repositories. We could make it generic to be able to switch between hg and git later. This module will remain in our repository.
scraper	Github (done)	Scraper for Gecko based applications from ftp.mozilla.org. We have to unify all the existing tools, e.g. getlatesttinderbox. This module will get its own github repository mozdownload.
testrun	n/a	Contains available test-runs to be used by the command line scripts

Proposal

Inherit from Mozrunner with Mozmill object composition

Mozmill 2.0 essentially has the functionality of 1.5 and yet the solutions are not trivial. This issue stems from the fact that TestRun currently inherits implicitly (via object composition) from Mozmill CLI. It enforces us to write TestRun in such a way that we are trying to keep the structure consistent with the CLI even though program flow and the structure is very different between 1.5 and 2.0.

Inheriting from Mozrunner is viable as we can use Mozrunner for Mozilla applications and use Mozmill's API as an object. These two modules will offer the same control that the CLI offers without the maintainability issue.

It is possible to inherit from Mozmill API with Mozrunner object composition or use multiple inheritance, but these relationships are less logical. The TestRun module should be a Mozrunner instance logically due to the fact that it handles the preparation and configuration of tests and the Mozilla application. Another reason would be because it helps to be able to override the init in Mozrunner as we do not want to use any of the Mozrunner parser options.

Changes to existing methods/new additions

The impact of removing Mozmill CLI will affect the following methods within TestRun:

• __init__
  • Added: logging capability
• _generate_custom_report
• prepare_binary
• prepare_repository
• prepare_tests
• run_tests
• run
• update_report

These methods will offer the same functionality to the consumer, but will be changed to offer support for Mozmill 2.

There will also be three new methods:

• get_report
• send_report(report_url)
• get_mozmill_version

These two reporting methods offers the TestRun the ability to get and send reports since this concept is very different in Mozmill 1.5 and 2.0. get_mozmill_version will use the Mozmill's metadata to return the version number. This might be useful for classes that inherit TestRun to be able to not run for specific versions.

Another possible feature is to have a constraint "allow_legacy_api" that will prevent systems with the Mozmill 1.5 API to run the tests.

Verification

To verify that the implementation is correctly working for Mozmill 1.5:

1. The results of an older TestRun must match the new one for all the current restart and non-restart tests
2. Ensure that the results are reported correctly

Likewise, for Mozmill 2.0:

1. Results of Mozmill 1.5 API restart and non-restart tests are the same for both Mozmill 2.0 and Mozmill 1.5
2. Ensure that tests using only Mozmill 2.0 API are working
3. Ensure that the new features (e.g. screenshot) are working by creating simple tests
4. Ensure that the results are reported correctly

Resources

• [Posts and Discussions]
• Presentations
• FAQ