QA/Automation/Projects/Mozmill Automation/Mozmill2Scripts: Difference between revisions
| Line 94: | Line 94: | ||
These methods will offer the same functionality to the consumer, but will be changed to offer support for Mozmill 2. | These methods will offer the same functionality to the consumer, but will be changed to offer support for Mozmill 2. | ||
There will also be | There will also be three new methods: | ||
* get_report | * get_report | ||
* send_report | * send_report(report_url) | ||
* get_mozmill_version | |||
These two methods offers the TestRun the ability to get and send reports since this concept is very different in Mozmill 1.5 and 2.0. | These two reporting methods offers the TestRun the ability to get and send reports since this concept is very different in Mozmill 1.5 and 2.0. | ||
get_mozmill_version will use the Mozmill's metadata to return the version number. This might be useful for classes that inherit TestRun to be able to not run for specific versions. | |||
Another possible feature is to have a constraint "allow_legacy_api" that will prevent systems with the Mozmill 1.5 API to run the tests. | |||
=== Verification === | |||
To verify that the implementation is correctly working for Mozmill 1.5: | |||
# The results of an older TestRun must match the new one for all the current restart and non-restart tests | |||
# Ensure that the results are reported correctly | |||
Likewise, for Mozmill 2.0: | |||
# Results of Mozmill 1.5 API restart and non-restart tests are the same for both Mozmill 2.0 and Mozmill 1.5 | |||
# Ensure that tests using only Mozmill 2.0 API are working | |||
# Ensure that the new features (e.g. screenshot) are working by creating simple tests | |||
# Ensure that the results are reported correctly | |||
==Resources== | ==Resources== | ||
Revision as of 23:35, 13 March 2012
Overview
| Name: | MozMill 2 Script Updates |
| Leads: | David Guo |
| Contributors: | TBD |
| Pivotal: | |
| Bugzilla: | https://bugzilla.mozilla.org/show_bug.cgi?id=732134 |
| Other Docs: | Index |
Goal History
| Period | Status | Goal |
| 2012 Q1 | [NEW] |
Requirements
- Remove Mozmill wrapper dependency
- Remove Mozmill CLI dependency
- All past tests must be able to be run on both Mozmill 2.0 and 1.5
- Must be compatible with Thunderbird and Firefox
- Must be maintainable
Project Milestones
| Milestone | Status | Description |
| Remove dependency of Mozmill wrapper | Complete; Implementation under review | Strategy: https://etherpad.mozilla.org/mozmill-wrapper-removal, Patch: https://bugzilla.mozilla.org/show_bug.cgi?id=732134 |
| Determine if current automation scripts can adapt to Mozmill 2.0 infrastructure | Complete | The conclusion was that there are no viable options without rewriting some parts of TestRun; Detailed information: https://etherpad.mozilla.org/mozmill2strategydraft |
| Draft a spec and design the TestRun module rewrite | Complete; Needs review | This effort will outline how TestRun will look inheriting from Mozrunner opposed to MozmillCLI |
| Implement the new TestRun | Not started | This effort will swap the Mozmill CLI inheritance with Mozrunner |
| Refactor and testing | Not started | After implementation is complete, the new TestRun module will be tested against previous tests and possible ways to refactor will be looked at |
Architecture
| Component | Description |
| TestRun | This class is to be used by the command line and will run a set of tests from a repository, and then send the report to a database. |
Proposal
Inherit from Mozrunner with Mozmill object composition
Mozmill 2.0 essentially has the functionality of 1.5 and yet the solutions are not trivial. This issue stems from the fact that TestRun currently inherits implicitly (via object composition) from Mozmill CLI. It enforces us to write TestRun in such a way that we are trying to keep the structure consistent with the CLI even though program flow and the structure is very different between 1.5 and 2.0.
Inheriting from Mozrunner is viable as we can use Mozrunner for Mozilla applications and use Mozmill's API as an object. These two modules will offer the same control that the CLI offers without the maintainability issue.
It is possible to inherit from Mozmill API with Mozrunner object composition or use multiple inheritance, but these relationships are less logical. The TestRun module should be a Mozrunner instance logically due to the fact that it handles the preparation and configuration of tests and the Mozilla application. Another reason would be because it helps to be able to override the init in Mozrunner as we do not want to use any of the Mozrunner parser options.
Changes to existing methods/new additions
The impact of removing Mozmill CLI will affect the following methods within TestRun:
- __init__
- Added: logging capability
- _generate_custom_report
- prepare_binary
- prepare_repository
- prepare_tests
- run_tests
- run
- update_report
These methods will offer the same functionality to the consumer, but will be changed to offer support for Mozmill 2.
There will also be three new methods:
- get_report
- send_report(report_url)
- get_mozmill_version
These two reporting methods offers the TestRun the ability to get and send reports since this concept is very different in Mozmill 1.5 and 2.0. get_mozmill_version will use the Mozmill's metadata to return the version number. This might be useful for classes that inherit TestRun to be able to not run for specific versions.
Another possible feature is to have a constraint "allow_legacy_api" that will prevent systems with the Mozmill 1.5 API to run the tests.
Verification
To verify that the implementation is correctly working for Mozmill 1.5:
- The results of an older TestRun must match the new one for all the current restart and non-restart tests
- Ensure that the results are reported correctly
Likewise, for Mozmill 2.0:
- Results of Mozmill 1.5 API restart and non-restart tests are the same for both Mozmill 2.0 and Mozmill 1.5
- Ensure that tests using only Mozmill 2.0 API are working
- Ensure that the new features (e.g. screenshot) are working by creating simple tests
- Ensure that the results are reported correctly