MozillaWiki

QA/Automation/Projects/Mozmill Automation: Difference between revisions

Page Discussion

QA/Automation/Projects/Mozmill Automation (view source)

Revision as of 22:59, 30 September 2011

449 bytes added ,  30 September 2011
no edit summary
No edit summary
Line 1: Line 1:
=Overview=
__NOTOC__
{|  
 
  | style="width: 33%" |
==Overview==
  | style="width: 33%" |
Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those are getting run on any platform, and also for localized builds of Firefox.
 
{| style="width: 90%; margin: 0 0 1em 1em" |
  | style="width: 20%" |
  | style="width: 80%" |
  |- valign="top"
  |- valign="top"
  | '''Lead:'''
  | '''Name:'''
  | [mailto:hskupin@mozilla.com Henrik Skupin]
  | Mozmill Automation
  |- valign="top"
  |- valign="top"
  | '''Co-workers:'''
  | '''Leads:'''
  | Al Billings, Anthony Hughes
  | [mailto:hskupin@mozilla.com Henrik Skupin], [mailto:gmealer@mozilla.com Geo Mealer]
  |- valign="top"
  |- valign="top"
  | '''Dates:'''
  | '''Contributors:'''
  | No end date set
  | Dave Hunt
  |- valign="top"
  |- valign="top"
  | '''Status:'''
  | '''Tracker:'''
  | Creating and Enhancing Scripts
  | https://www.pivotaltracker.com/projects/298905
  |- valign="top"
  |- valign="top"
| '''Specs:'''
  | '''Repository:'''  
| [http://docs.google.com/Doc?docid=0AVroormSQgiRZDljejN3a18xNmNrdmY2a2Rn&hl=en Add-on test-run script]
  | http://hg.mozilla.org/qa/mozmill-automation/
|- valign="top"
  | '''Repository Location:'''
  | [http://hg.mozilla.org/qa/mozmill-automation/file/tip Automated Test-run Scripts]
  |- valign="top"
  |- valign="top"
  | '''Tracking Bug:'''
  | '''Etherpad:'''
  | {{bug|562639}}
  | http://etherpad.mozilla.com:9000/qas-mozmill-automation
|}
|}


=Excerpt=
==Goals for Q3/11==
Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We plan to have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those will have to be run on any platform and also against localized builds of Firefox.
TBD.


=Project details=
{| class="fullwidth-table" |
The way to reach the state when everything is automated is long can can be divided into the following sub-projects:
| style="background:#EFEFEF; width: 15%" | Status
| style="background:#EFEFEF; width: 25%" | Task
| style="background:#EFEFEF; width: 60%" | Description
|- valign="top"
|
|
|
|}


* Setting up a machine which can be used for automated tests
==Sub Projects==
* Creation of scripts which cover all automation aspects
{| class="fullwidth-table" |
* Running the automation for release builds
| style="background:#EFEFEF; width: 25%" | Sub Project
 
| style="background:#EFEFEF; width: 55%" | Description
==Machine for Automated Testing==
|- valign="top"
A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.
| [[QA/Automation_Services/Projects/Mozmill_Shared_Modules/|QA/Automation_Services/Projects/Mozmill_Shared_Modules/]]
 
|  
This sub-project has been finished. All requirements can be read on {{bug|537840}}
|}
 
==Automation Execution==
Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.
 
==Automation Scripts==
The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:
 
* Script for Downloading builds from the FTP Server ({{bug|528064}})
* Script for Smoketests, BFT, and FFT Test-runs ({{bug|563523}})
* Script for Software Update Test-runs ({{bug|564539}})
* Script for Add-ons Test-run ({{bug|562445}})
* Script for L10n Test-run ({{bug|565196}})
* Script for Accessibility Test-run ({{bug|565201}})
 
===Downloading builds from the FTP Server===
For a complete automation of our Mozmill tests for release testing, a download script is necessary which will fetch all the builds to test from the FTP server and store those locally. The work is being tracked on {{bug|528064}}.
 
===Smoketests, BFT, and FFT Test-run===
To execute all existing Mozmill tests which have been written for the Smoketest, BFT, and FFT testgroup of Litmus this script is useful. Once it has been called it should automatically execute the steps below:
 
* Install a build (if requested)
* Run the tests from all those three testgroups
* Uninstall the build (if requested)
 
The work is being  tracked on {{bug|563523}}.
 
===Software Update Test-run===
To run the software update tests automatically a test-run script is required which will take care of any aspect of the tests to be run. Once it has been called it should automatically execute the steps below:


* Install a build (if requested)
==Details==
* Create a backup of the build
Whenever you want to start to automate a new BFT/FFT subgroup on Litmus, you should first think about having a shared module in place. Creating a list of requirements should be the first step you will have to do as early as possible in the whole process. That will allow us to help you or even write the shared module on our own. Once it has been done, you can start writing Mozmill tests for that subgroup.
* Run the partial or complete software update tests
* Restore the backup
* Run the fallback software update tests
* Restore the backup
* Run background software update tests
* Uninstall the build (if requested)


A script with this feature set has already been created on {{bug|504653}} and is able to execute all the steps except step 6. The reason is that we do not have any automated tests for background software updates yet.
===Identifying Requirements===
This part is probably not trivial if you are doing it the first time. That's why we want to offer some help for the requirements definition process. By following the steps below you will be able to put a good list of items together.  


===Add-on Test-run===
Before you start with the list below, make sure that there hasn't been already made a request. Therefore check the BFT table of the [http://spreadsheets.google.com/ccc?key=pAP5Y5AH3-Tl-wRoNgBujUQ&pli=1 Mozmill spreadsheet] and also all open requests for shared modules. If your request cannot be found, you can start to identify the requirements.  
The script for the add-ons test-run simplifies the work which would have to be done to run Mozmill tests against available add-ons. Once it has been called it should automatically execute the steps below:


* Install a build (if requested)
1. Each of the tests will interact with a couple of ui elements. All those elements can be accessed in different ways. The easiest way is definitely its node id. But there are probably more elements which can't be reached by that method and requires a complex lookup expression. As a test writer you don't wanna be confronted with such a research task because it requires knowledge about the DOM and tools like the [https://addons.mozilla.org/de/firefox/addon/6622 DOM inspector]. Instead you want to simply use those elements and create sequential steps of actions. To be able to do that, the shared module has to give access to all elements in the same simple manner by only specifying the elements name and eventually an index. As result the getElement function has been added to all shared modules and has to be used to retrieve elements.
* Perform Mozmill tests against any available add-on
** Download the specified version of the add-on
** Start Firefox with the extension installed
** Run all available tests
** Clean-up the system
* Uninstall the build (if requested)


The work is being  tracked on {{bug|562445}}.
2. Tests from the same subgroup normally have similar actions which have to be performed to get into a given state. Those single steps shouldn't be performed again and again by the tests itself. Instead a function should be created in the shared module which combines those steps in a single call.  


===L10n Test-run===
3. Sometimes the new shared module can have dependencies to other shared modules. Take the AddonsAPI as an example. It reads and writes some preferences and needs functions from the PrefsAPI.
With the creation of [[QA/Mozmill_Test_Automation/L10n_Tests|l10n tests]] we will be able to perform specific tests for localizers. Once it has been called it should automatically execute the steps below:


* Install a build (if requested)
===Filing a Bug===
* Run all the l10n tests
To get the shared module implemented you will have to file a [https://bugzilla.mozilla.org/enter_bug.cgi?component=Mozmill Shared Modules&form_name=enter_bug&op_sys=All&product=Mozilla QA&rep_platform=Al&short_desc=Implement shared module for %25Feature%25 new bug]. Use a clear summary and add the requirements you have compiled in the former chapter.
* Uninstall the build (if requested)


The work is being  tracked on {{bug|565196}}.
===Implementation===
The creation of shared modules is not as easy as creating tests, but it's more challenging because you really have to think in an abstract way. If you haven't written a shared module yet, please check our [http://hg.mozilla.org/qa/mozmill-tests/file/default/lib/ existing shared modules]. Those can give you an insight how the code has to look like. You should also obey the [https://developer.mozilla.org/en/Mozmill_Tests#Writing_Mozmill_Tests test writing guidelines].  


===Accessibility Test-run===
While creating the shared module you should immediately check if the functions you have added are working as expected. Therefore you can create tests under [http://hg.mozilla.org/qa/mozmill-tests/file/default/lib/tests lib/tests] with the same name as the shared module or the class itself. The [https://developer.mozilla.org/en/Mozmill_Tests/Shared_Modules#How_to_use_a_Shared_Module shared module documentation] explains how to include the shared module into a test.  
With the creation of [[QA/Mozmill_Test_Automation/Accessibility_Tests|accessibility tests]] we will be able to perform specific tests for keyboard navigation. Once it has been called it should automatically execute the steps below:


* Install a build (if requested)
Before the shared module can be checked-in a review is necessary. Therefore create a patch and ask for review. Details about that process can be found on [https://developer.mozilla.org/en/Mozmill_Tests#The_Review_Process MDC].
* Run all the accessibility tests
* Uninstall the build (if requested)


The work is being  tracked on {{bug|565201}}.
==Resources==
* [[/Public|Public Posts and Discussions]]
* [[/Presentations|Presentations]]
* [[/FAQ|FAQ]]
* [[/Contribution|How to Contribute]]
Whimboo
canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,714

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/353665"