QA/Automation/Projects/Mozmill Automation: Difference between revisions

Revision as of 23:03, 30 September 2011

Overview

Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those are getting run on any platform, and also for localized builds of Firefox.


Name:	Mozmill Automation
Leads:	Henrik Skupin, Geo Mealer
Contributors:	Dave Hunt
Tracker:	https://www.pivotaltracker.com/projects/298905
Repository:	http://hg.mozilla.org/qa/mozmill-automation/
Etherpad:	http://etherpad.mozilla.com:9000/qas-mozmill-automation

Goals for Q3/11

TBD.

Status	Task	Description

Sub Projects

Sub Project	Description
API refactor

Details

The way to reach the state when everything is automated is long can can be divided into the following sub-projects:

Setting up a machine which can be used for automated tests
Creation of scripts which cover all automation aspects
Running the automation for release builds

Machine for Automated Testing

A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.

This sub-project has been finished. All requirements can be read on bug 537840

Automation Execution

Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.

Automation Scripts

The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:

Script for Downloading builds from the FTP Server (bug 528064)
Script for Smoketests, BFT, and FFT Test-runs (bug 563523)
Script for Software Update Test-runs (bug 564539)
Script for Add-ons Test-run (bug 562445)
Script for L10n Test-run (bug 565196)
Script for Accessibility Test-run (bug 565201)

Downloading builds from the FTP Server

For a complete automation of our Mozmill tests for release testing, a download script is necessary which will fetch all the builds to test from the FTP server and store those locally. The work is being tracked on bug 528064.

Smoketests, BFT, and FFT Test-run

To execute all existing Mozmill tests which have been written for the Smoketest, BFT, and FFT testgroup of Litmus this script is useful. Once it has been called it should automatically execute the steps below:

Install a build (if requested)
Run the tests from all those three testgroups
Uninstall the build (if requested)

The work is being tracked on bug 563523.

Software Update Test-run

To run the software update tests automatically a test-run script is required which will take care of any aspect of the tests to be run. Once it has been called it should automatically execute the steps below:

Install a build (if requested)
Create a backup of the build
Run the partial or complete software update tests
Restore the backup
Run the fallback software update tests
Restore the backup
Run background software update tests
Uninstall the build (if requested)

A script with this feature set has already been created on bug 504653 and is able to execute all the steps except step 6. The reason is that we do not have any automated tests for background software updates yet.

Add-on Test-run

The script for the add-ons test-run simplifies the work which would have to be done to run Mozmill tests against available add-ons. Once it has been called it should automatically execute the steps below:

Install a build (if requested)
Perform Mozmill tests against any available add-on
- Download the specified version of the add-on
- Start Firefox with the extension installed
- Run all available tests
- Clean-up the system
Uninstall the build (if requested)

The work is being tracked on bug 562445.

L10n Test-run

With the creation of l10n tests we will be able to perform specific tests for localizers. Once it has been called it should automatically execute the steps below:

Install a build (if requested)
Run all the l10n tests
Uninstall the build (if requested)

The work is being tracked on bug 565196.

Accessibility Test-run

With the creation of accessibility tests we will be able to perform specific tests for keyboard navigation. Once it has been called it should automatically execute the steps below:

Install a build (if requested)
Run all the accessibility tests
Uninstall the build (if requested)

The work is being tracked on bug 565201.

@@ Line 1: / Line 1: @@
-=Overview=
+__NOTOC__
-{|
-  | style="width: 33%" |
+==Overview==
-  | style="width: 33%" |
+Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those are getting run on any platform, and also for localized builds of Firefox.
+{| style="width: 90%; margin: 0 0 1em 1em" |
+  | style="width: 20%" |
+  | style="width: 80%" |
+ |- valign="top"
+ | '''Name:'''
+ | Mozmill Automation
   |- valign="top"
-  | '''Lead:'''
+  | '''Leads:'''
-  | [mailto:hskupin@mozilla.com Henrik Skupin]
+  | [mailto:hskupin@mozilla.com Henrik Skupin], [mailto:gmealer@mozilla.com Geo Mealer]
   |- valign="top"
-  | '''Co-workers:'''
+  | '''Contributors:'''
-  | Al Billings, Anthony Hughes
+  | Dave Hunt
   |- valign="top"
-  | '''Dates:'''
+  | '''Tracker:'''
-  | No end date set
+  | https://www.pivotaltracker.com/projects/298905
   |- valign="top"
-  | '''Status:'''
+  | '''Repository:'''
-  | Creating and Enhancing Scripts
+  | http://hg.mozilla.org/qa/mozmill-automation/
   |- valign="top"
-  | '''Specs:'''
+  | '''Etherpad:'''
-  | [http://docs.google.com/Doc?docid=0AVroormSQgiRZDljejN3a18xNmNrdmY2a2Rn&hl=en Add-on test-run script]
+  | http://etherpad.mozilla.com:9000/qas-mozmill-automation
-|- valign="top"
+|}
-  | '''Repository Location:'''
-  | [http://hg.mozilla.org/qa/mozmill-automation/file/tip Automated Test-run Scripts]
+==Goals for Q3/11==
+TBD.
+{| class="fullwidth-table" |
+ | style="background:#EFEFEF; width: 15%" | Status
+  | style="background:#EFEFEF; width: 25%" | Task
+  | style="background:#EFEFEF; width: 60%" | Description
   |- valign="top"
-  | '''Tracking Bug:'''
+  |
-  | {{bug|562639}}
+  |
- |}
+ |
+|}
-=Excerpt=
+==Sub Projects==
-Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We plan to have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those will have to be run on any platform and also against localized builds of Firefox.
+{| class="fullwidth-table" |
+ | style="background:#EFEFEF; width: 25%" | Sub Project
+ | style="background:#EFEFEF; width: 55%" | Description
+ |- valign="top"
+ | [[QA/Automation_Services/Projects/Mozmill_Shared_Modules/API_Refactor|API refactor]]
+ |
+|}
-=Project details=
+==Details==
 The way to reach the state when everything is automated is long can can be divided into the following sub-projects:
@@ Line 36: / Line 56: @@
 * Running the automation for release builds
-==Machine for Automated Testing==
+===Machine for Automated Testing===
 A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.
 This sub-project has been finished. All requirements can be read on {{bug|537840}}
-==Automation Execution==
+===Automation Execution===
 Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.
-==Automation Scripts==
+===Automation Scripts===
 The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:
@@ Line 54: / Line 74: @@
 * Script for Accessibility Test-run ({{bug|565201}})
-===Downloading builds from the FTP Server===
+====Downloading builds from the FTP Server====
 For a complete automation of our Mozmill tests for release testing, a download script is necessary which will fetch all the builds to test from the FTP server and store those locally. The work is being tracked on {{bug|528064}}.
-===Smoketests, BFT, and FFT Test-run===
+====Smoketests, BFT, and FFT Test-run====
 To execute all existing Mozmill tests which have been written for the Smoketest, BFT, and FFT testgroup of Litmus this script is useful. Once it has been called it should automatically execute the steps below:
@@ Line 66: / Line 86: @@
 The work is being  tracked on {{bug|563523}}.
-===Software Update Test-run===
+====Software Update Test-run====
 To run the software update tests automatically a test-run script is required which will take care of any aspect of the tests to be run. Once it has been called it should automatically execute the steps below:
@@ Line 80: / Line 100: @@
 A script with this feature set has already been created on {{bug|504653}} and is able to execute all the steps except step 6. The reason is that we do not have any automated tests for background software updates yet.
-===Add-on Test-run===
+====Add-on Test-run====
 The script for the add-ons test-run simplifies the work which would have to be done to run Mozmill tests against available add-ons. Once it has been called it should automatically execute the steps below:
@@ Line 93: / Line 113: @@
 The work is being  tracked on {{bug|562445}}.
-===L10n Test-run===
+====L10n Test-run====
 With the creation of [[QA/Mozmill_Test_Automation/L10n_Tests|l10n tests]] we will be able to perform specific tests for localizers. Once it has been called it should automatically execute the steps below:
@@ Line 102: / Line 122: @@
 The work is being  tracked on {{bug|565196}}.
-===Accessibility Test-run===
+====Accessibility Test-run====
 With the creation of [[QA/Mozmill_Test_Automation/Accessibility_Tests|accessibility tests]] we will be able to perform specific tests for keyboard navigation. Once it has been called it should automatically execute the steps below: