MozillaWiki

QA/Automation/Projects/Mozmill Shared Modules/On Demand Testing: Difference between revisions

Page Discussion

QA/Automation/Projects/Mozmill Shared Modules/On Demand Testing (view source)

Revision as of 10:40, 13 January 2014

1,081 bytes added ,  13 January 2014
m
Whimboo moved page Auto-tools/Automation Development/Projects/Mozmill Shared Modules/On Demand Testing to QA/Automation/Projects/Mozmill Shared Modules/On Demand Testing: Team reorganization
 
(11 intermediate revisions by 2 users not shown)
Line 8: Line 8:
  |- valign="top"
  |- valign="top"
  | '''Co-workers:'''
  | '''Co-workers:'''
  | Henrik Skupin
  | Henrik Skupin, Al Billings
  |- valign="top"
  |- valign="top"
  | '''Dates:'''
  | '''Dates:'''
  | Completion at end of Q1 2011
  | Ongoing; Phase III to be completed end of Q3 2011
  |- valign="top"
  |- valign="top"
  | '''Status:'''
  | '''Status:'''
Line 17: Line 17:
  |- valign="top"
  |- valign="top"
  | '''Repository Location:'''
  | '''Repository Location:'''
  | TBD
  | http://hg.mozilla.org/qa/mozmill-automation/file/default/ondemand
  |- valign="top"
  |- valign="top"
  | '''Tracking Bug / Bug List:'''
  | '''Tracking Bug / Bug List:'''
Line 23: Line 23:
  |- valign="top"
  |- valign="top"
  | '''Documentation:'''
  | '''Documentation:'''
  | [https://wiki.mozilla.org/QA/Mozmill_Test_Automation/On_Demand_Update_Testing/Documentation wiki]
  | [http://pulse.mozilla.org/ Pulse], [https://wiki.mozilla.org/QA/Mozmill_Test_Automation/On_Demand_Update_Testing/Documentation Command line docs]
  |}
  |}


Line 47: Line 47:
# From the test plan, create a configuration file including the desired builds, platforms, and locales.
# From the test plan, create a configuration file including the desired builds, platforms, and locales.
# Run a single script that reads the configuration file, automatically downloads the desired builds, and stages them in the disk share.
# Run a single script that reads the configuration file, automatically downloads the desired builds, and stages them in the disk share.
# Run a single script that uses ssh to log into each target platform machine and executes the existing update testing script.
# Run a single script that uses pulse to kick off a process on each target platform machine and executes the existing update testing script.
# Look in brasstacks (our automation result reporting framework) for the update test results as they land.
# Look in brasstacks (our automation result reporting framework) for the update test results as they land.


We have already moved to putting the results in brasstacks. The balance of Phase I requires creation of two scripts:
===The Staging Script===
The staging script takes as input a configuration file containing a list of all desired platforms, builds, and locales.
 
The script expands the configuration file's lists into a comprehensive list of desired builds, and downloads into the staging area.
 
===The Execution Script===
The execution system consists of two parts, a test pusher and a test listener.


* The staging script
The test pusher defines the branch, update channel, and optionally the platform to be tested. The test listener runs on each desired platform; when a push for all platforms or for the selected platform is received, it launches the existing update test scripts.
* The execution script


===The Staging Script===
The existing update test scripts consume the directory tree created by the staging script to perform the desired total test run.
The staging script should take as input a configuration file containing a list of all desired platforms, builds, and locales.
 
==The Process after Phase II==
Phase I concentrates on improving the Phase I process in two ways:


It should also take an option which is the root of the staging file system.
# We've introduced a cluster concept so that multiple test clusters can be run without interfering with each other.
# The system now works with functional tests as well. In fact, this project page is basically obsolete and now needs to be replaced with "On Demand Release Testing," as the system can now handle all aspects.


The script will expand the configuration file's lists into a comprehensive list of desired builds, and download into the staging area.
We've also introduced a new component, a heartbeat emitter. This is a temporary workaround, because we found that rolling out Pulse listeners that receive only sporadic traffic causes their sockets to be closed out from under them. The heartbeat emitter is simply a traffic generator.


===The Execution Script===
==The Process after Phase III (In Progress)==
The execution script should take as input a configuration file containing a list of target execution platforms.
Phase III will be improving Phase II in one primary way:


The script will iterate through the list, logging into each platform and running the current update testing script. The current script already knows how to look in the staging area.
# The system will be kicked off from a web front end rather than via command-line


Login details will be handled via public/private key setup rather than name/PW to avoid having to write a secure way to encapsulate login info.
The primary gain here will be simplifying the process for those who aren't used to ssh and command-line tools. Our goal is to simplify the test run process to the point that a non-automation person can ramp up on how to do it within minutes.
Whimboo
canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,714

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/308670...887771"