QA/Automation/Projects/Mozmill Automation/On Demand Test Framework: Difference between revisions
| Line 87: | Line 87: | ||
If an off-the-shelf CI tool like Jenkins turns out be appropriate for our needs, it may subsume the Dashboard and Executor components. | If an off-the-shelf CI tool like Jenkins turns out be appropriate for our needs, it may subsume the Dashboard and Executor components. | ||
The question of which module controls/watchdogs the individual Test Machines performing the run has not been answered yet. Further analysis will show whether that's better as part of the Executor, Provisioner, whether the machine can run relatively autonomously once it's been initialized, or whether a new | The question of which module controls/watchdogs the individual Test Machines performing the run has not been answered yet. Further analysis will show whether that's better as part of the Executor, Provisioner, whether the machine can run relatively autonomously once it's been initialized, or whether a new component is needed. | ||
In addition to these, Status formats must be defined where they exist in the architecture. It is unknown yet as to which module will define these, or whether they will be significant enough to require subprojects. | In addition to these, Status formats must be defined where they exist in the architecture. It is unknown yet as to which module will define these, or whether they will be significant enough to require subprojects. | ||
Revision as of 01:47, 1 November 2011
Overview
(need to expand this, this is not at all final wording)
The On-Demand Test Framework will be a full-service execution framework for Mozmill and perhaps eventually other tests. It will allow execution of tests across a fabric of machines. Eventually, it will include auto-provisioning that fabric and features to allow parallel execution.
| Name: | On-Demand Test Framework |
| Leads: | Geo Mealer, Henrik Skupin |
| Contributors: | TBD |
| Pivotal: | Tracker |
| Bugzilla: | Tracking Bug |
Goal History
| Period | Status | Goal |
| 2011 Q4 | [ON TRACK] | Implementation of a remote mechanism to trigger Mozmill functional and update tests for Firefox releases |
Project Milestones
| Milestone | Status | Description |
| Basic Execution | Design Stage
[ON TRACK] |
System allows execution of test-run demands against standing machines. |
| Auto-Provisioning | Not started | System automatically provisions and/or resets virtual machines |
| Parallelism | Not started | System knows how to slice up a test run more finely than per-platform |
Architecture
The Trigger and Config Generator components are a proposed part of integrating the CI goal into this system by automatically generating a Demand, and will not be covered in this project.
| Component | Description |
| Demand | User-facing interface for demanding a one-time test run |
| Config | Standardized configuration document for describing a test run |
| Dashboard | User-facing interface for tracking run status |
| Executor | Kicks off and centrally monitors the test run |
| Provisioner | Initializes appropriate machines for execution |
| Machine | Environment on target machines for remote execution |
If an off-the-shelf CI tool like Jenkins turns out be appropriate for our needs, it may subsume the Dashboard and Executor components.
The question of which module controls/watchdogs the individual Test Machines performing the run has not been answered yet. Further analysis will show whether that's better as part of the Executor, Provisioner, whether the machine can run relatively autonomously once it's been initialized, or whether a new component is needed.
In addition to these, Status formats must be defined where they exist in the architecture. It is unknown yet as to which module will define these, or whether they will be significant enough to require subprojects.