QA/Automation/Projects/Addons/MozmillCrowd: Difference between revisions

From MozillaWiki
< QA‎ | Automation‎ | Projects‎ | Addons
Jump to navigation Jump to search
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
=Overview=
__NOTOC__
{|  
==Overview==
  | style="width: 33%" |
Running Mozmill tests should be as easy as possible and doable by any interested tester. No complicated setup routines or pre-requisites should stop anyone by successfully running our existing Mozmill tests in their Firefox browser. As result a rich set of results, which will be collected from testers world-wide, can be collected and used to nail down regressions or to find problems related to the geographical position or installed locale of Firefox.
  | style="width: 33%" |
 
{| style="width: 100%; margin-bottom: 2em" |
  | style="width: 25%" |
  | style="width: 75%" |
  |- valign="top"
  |- valign="top"
  | '''Lead:'''
  | '''Name:'''
  | [mailto:hskupin@mozilla.com Henrik Skupin]
  | Mozmill Crowd - Crowd-sourced functional Testing with Mozmill
  |- valign="top"
  |- valign="top"
  | '''Co-workers:'''
  | '''Leads:'''
  | n/a (help wanted)
  | [mailto:hskupin@mozilla.com Henrik Skupin], [mailto:atrain@mozilla.com Aaron Train]
  |- valign="top"
  |- valign="top"
  | '''Dates:'''
  | '''Team Assistance:'''
  | v0.1 by end of Q2/10
  | n/a
  |- valign="top"
  |- valign="top"
  | '''Status:'''
  | '''Tracker:'''
  | Collecting requirements
  | https://www.pivotaltracker.com/projects/285911
|- valign="top"
|}
| '''Repository Location:'''
| [http://github.com/whimboo/mozmill-cloud http://github.com/whimboo/mozmill-cloud]
|- valign="top"
| '''Tracking Bug:'''
| [https://bugzilla.mozilla.org/show_bug.cgi?id=564648 Bug 564648]
|}
 
=Excerpt=
Running Mozmill tests should be as easy as possible and doable by any interested party. No complicated setup routines or pre-requisites should stop anyone by successfully running our existing Mozmill tests in their Firefox browser. As result a rich set of results, which will be collected from testers world-wide, will be stored on our results server and used to nail down regressions or to find problems related to your geographical position or installed language pack.
 
=Project Details=
Only running Mozmill tests on buildbot or on the release-test machine in the QA lab is not sufficient to find misbehavior's of Firefox. A more wide-spread test-run is necessary. Such a possibility will be the result of the cloud testing project. An extension has to be developed which will automatically prepare the system, executes the test-run, and send the results to our report server for analysis.
 
==The Extension==
The easiest way to add additional features to Firefox is to install an extension. The same way will be used for this project. The extension, which will be offered on [https://addons.mozilla.org/ addons.mozilla.org], will add it's own toolbar right below of the existing ones. With a dropdown field the user will be able to select the wanted test-run, e.g. normal tests, update tests, or l10n tests. By clicking the 'Run' button the test-run gets executed.
 
==Preparing the System==
To run Mozmill tests, the system has to be prepared to match a specific set of requirements. [https://developer.mozilla.org/en/Mozmill#The_Command_Line_Client The steps] which are necessary aren't the easiest ones and can cause trouble, especially when you are doing it for the first time.
 
That's why the extension has to create a virtual environment, which will contain all needed applications and tools. The following steps will be necessary:
 
* Retrieving the Python interpreter (or bundle it with the extension)
* Create a virtual Python environment
* Install the latest release of Mozmill
* Retrieve the automation scripts


==Executing Mozmill Tests==
==Platform==
Given by the selected dropdown entry on the toolbar the Python script of the related test-run gets executed. It will perform the following actions:
* [[/Documentation|Documentation]]
* [http://etherpad.mozilla.com:9000/qa-auto-mozmill-crowd Etherpad] for brainstorming and discussions
* [[/FAQ|FAQ]]
* [[/Presentations|Presentations]]


* Cloning the Mozmill tests from the [http://hg.mozilla.org/qa/mozmill-tests/ mozmill-tests repository]
==Production==
* Updating the tests branch to the Gecko branch of the currently running version of Firefox
* [[/Roadmap|Product Roadmap]]
* Starting another instance of Firefox
* [[/Release|Release Process]]
* Execute all existent tests
* [[/Contribution|How to Contribute]]


==Results Analysis==
==Resources==
Once a test-run has been finished the results can be uploaded to our [http://brasstacks.mozilla.com/couchdb/mozmill/_design/reports/_list/summary/summary report server]. There they can be examined and checked for new failures. A [https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Dashboard web dashboard] will be created to offer the most valuable information.

Latest revision as of 10:40, 13 January 2014

Overview

Running Mozmill tests should be as easy as possible and doable by any interested tester. No complicated setup routines or pre-requisites should stop anyone by successfully running our existing Mozmill tests in their Firefox browser. As result a rich set of results, which will be collected from testers world-wide, can be collected and used to nail down regressions or to find problems related to the geographical position or installed locale of Firefox.

Name: Mozmill Crowd - Crowd-sourced functional Testing with Mozmill
Leads: Henrik Skupin, Aaron Train
Team Assistance: n/a
Tracker: https://www.pivotaltracker.com/projects/285911

Platform

Production

Resources