MozillaWiki

QA/Automation/Projects/Mozmill Automation: Difference between revisions

Page Discussion

QA/Automation/Projects/Mozmill Automation (view source)

Revision as of 21:28, 10 September 2014

3,024 bytes added ,  10 September 2014
→‎Sub Projects
 
(34 intermediate revisions by 3 users not shown)
Line 1: Line 1:
=Overview=
__NOTOC__
{|  
 
  | style="width: 33%" |
==Overview==
  | style="width: 33%" |
Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those are getting run on any platform, and also for localized builds of Firefox.
 
{| style="width: 90%; margin: 0 0 1em 1em" |
  | style="width: 20%" |
  | style="width: 80%" |
|- valign="top"
| '''Name:'''
| Mozmill Automation
  |- valign="top"
  |- valign="top"
  | '''Lead:'''
  | '''Leads:'''
  | [mailto:hskupin@mozilla.com Henrik Skupin]
  | [mailto:hskupin@mozilla.com Henrik Skupin]
  |- valign="top"
  |- valign="top"
  | '''Co-workers:'''
  | '''Contributors:'''
  | Al Billings, Anthony Hughes
| n/a
|- valign="top"
| '''Etherpad:'''
  | http://etherpad.mozilla.com:9000/qa-mozmill-automation
|}
 
==Sub Projects==
{| class="fullwidth-table" |
| style="background:#EFEFEF; width: 25%" | Sub Project
| style="background:#EFEFEF; width: 15%" | Status
| style="background:#EFEFEF; width: 40%" | Description
  |- valign="top"
  |- valign="top"
  | '''Dates:'''
  | [[QA/Automation/Projects/Mozmill_Automation/Mozmill_CI|Mozmill CI]]
  | No end date set
| Active
  | Perform fully automated Mozmill testruns via Jenkins CI
  |- valign="top"
  |- valign="top"
  | '''Status:'''
  | [[QA/Automation/Projects/Mozmill_Automation/Puppet|PuppetAgain]]
  | Creating and Enhancing Scripts
| Active
  | Puppetizing Mozmill CI machines
  |- valign="top"
  |- valign="top"
  | '''Specs:'''
  | [[QA/Automation/Projects/Mozmill_Automation/Environment|Mozmill Environment]]  
| [http://docs.google.com/Doc?docid=0AVroormSQgiRZDljejN3a18xNmNrdmY2a2Rn&hl=en Add-on test-run script]
| Active
|- valign="top"
| Supply a pre-configured environment for Mozmill tests execution
  | '''Repository Location:'''
|- valign="top"  
| [http://hg.mozilla.org/qa/mozmill-automation/file/tip Automated Test-run Scripts]]
  | [[QA/Automation/Projects/Mozmill_Automation/Mozmill_CI_Configuration_Generator|Mozmill CI Configuration Generator]]  
| Active
| Application to generate valid configuration files for Mozmill CI on-demand jobs
  |- valign="top"
  |- valign="top"
  | '''Tracking Bug:'''
  | <strike>[[QA/Automation/Projects/Mozmill_Automation/Mozmill2Scripts|Mozmill 2 scripts]]</strike>
  | {{bug|562639}}
| Finished
  |}
  | Update automation scripts for Mozmill to support Mozmill 2.0
|- valign="top"
  | <strike>[[QA/Automation/Projects/Mozmill_Automation/On_Demand_Test_Framework|On-Demand Test Framework]]</strike>
| Finished
| <strike>Implementation of a remote mechanism to trigger Mozmill functional and update tests for Firefox releases</strike>
|- valign="top"
| <strike>[[QA/Automation/Projects/Mozmill_Automation/Triggered_l10n_Test-runs|Triggered l10n Test-runs]]</strike>
| Finished
| <strike>Perform l10n related tests triggered by landing of localization changes</strike>
|- valign="top"
| <strike>[[QA/Automation/Projects/Mozmill_Automation/On_Demand_Testing|On-Demand Testing]]</strike>
| Finished
| <strike>Streamline how testing is performed on releases</strike>
|}


=Excerpt=
==Details==
Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We plan to have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those will have to be run on any platform and also against localized builds of Firefox.
 
=Project details=
The way to reach the state when everything is automated is long can can be divided into the following sub-projects:
The way to reach the state when everything is automated is long can can be divided into the following sub-projects:


Line 36: Line 67:
* Running the automation for release builds
* Running the automation for release builds


==Machine for Automated Testing==
===Machine for Automated Testing===
A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.
A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.


This sub-project has been finished. All requirements can be read on {{bug|537840}}
This sub-project has been finished. All requirements can be read on {{bug|537840}}


==Automation Execution==
===Automation Execution===
Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.
Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.


==Automation Scripts==
===Automation Scripts===
The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:
The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:


* Download script to fetch builds from the FTP server ({{bug|528064}}
* Script for Downloading builds from the FTP Server ({{bug|528064}})
* Script for Smoketests, BFT, and FFT Test-runs
* Script for Smoketests, BFT, and FFT Test-runs ({{bug|563523}})
* Script for Software Update Test-runs ({{bug|564539}})
* Script for Software Update Test-runs ({{bug|564539}})
* Script for Add-ons Test-run ({{bug|562445}})
* Script for Add-ons Test-run ({{bug|562445}})
* Script for L10n Test-run ({{bug|565196}})
* Script for Accessibility Test-run ({{bug|565201}})


===Software Update Script===
====Downloading builds from the FTP Server====
For a complete automation of our Mozmill tests for release testing, a download script is necessary which will fetch all the builds to test from the FTP server and store those locally. The work is being tracked on {{bug|528064}}.
 
====Smoketests, BFT, and FFT Test-run====
To execute all existing Mozmill tests which have been written for the Smoketest, BFT, and FFT testgroup of Litmus this script is useful. Once it has been called it should automatically execute the steps below:
 
* Install a build (if requested)
* Run the tests from all those three testgroups
* Uninstall the build (if requested)
 
The work is being  tracked on {{bug|563523}}.
 
====Software Update Test-run====
To run the software update tests automatically a test-run script is required which will take care of any aspect of the tests to be run. Once it has been called it should automatically execute the steps below:
To run the software update tests automatically a test-run script is required which will take care of any aspect of the tests to be run. Once it has been called it should automatically execute the steps below:


* Install a build
* Install a build (if requested)
* Create a backup of the build
* Run the partial or complete software update tests
* Run the partial or complete software update tests
* Reinstall the build
* Restore the backup
* Run the fallback software update tests
* Run the fallback software update tests
* Reinstall the build
* Restore the backup
* Run background software update tests
* Run background software update tests
* Uninstall the build
* Uninstall the build (if requested)


A script with this feature set has already been created on {{bug|504653}} and is able to execute all the steps except step 6. The reason is that we do not have any automated tests for background software updates yet.
A script with this feature set has already been created on {{bug|504653}} and is able to execute all the steps except step 6. The reason is that we do not have any automated tests for background software updates yet.


===Script for Add-on Test-runs===
====Add-on Test-run====
While you would also be able to run tests against extensions directly with Mozmill itself, you would have to download the extension first and specify it as argument to the Mozmill command line client. Those steps can be simplified and will result in a new test-run script for add-ons.  
The script for the add-ons test-run simplifies the work which would have to be done to run Mozmill tests against available add-ons. Once it has been called it should automatically execute the steps below:


It will become a Python script which can be used to run tests against all or only a particular extension. Once it has been called it should automatically execute the steps below:
* Install a build (if requested)
 
* Perform Mozmill tests against any available add-on
* Perform tests against each listed extension
** Download the specified version of the add-on
** Download the latest version from the specified URL
** Start Firefox with the extension installed
** Start Firefox with the extension installed
** Run all available tests
** Run all available tests
** Clean-up the system
** Clean-up the system
* Uninstall the build (if requested)
The work is being  tracked on {{bug|562445}}.
====L10n Test-run====
With the creation of [[QA/Mozmill_Test_Automation/L10n_Tests|l10n tests]] we will be able to perform specific tests for localizers. Once it has been called it should automatically execute the steps below:
* Install a build (if requested)
* Run all the l10n tests
* Uninstall the build (if requested)
The work is being  tracked on {{bug|565196}}.
====Accessibility Test-run====
With the creation of [[QA/Mozmill_Test_Automation/Accessibility_Tests|accessibility tests]] we will be able to perform specific tests for keyboard navigation. Once it has been called it should automatically execute the steps below:
* Install a build (if requested)
* Run all the accessibility tests
* Uninstall the build (if requested)
The work is being  tracked on {{bug|565201}}.
Whimboo
canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,714

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/226768...1014090"