QA/Automation/Projects/Mozmill Shared Modules/On Demand Testing: Difference between revisions
| Line 11: | Line 11: | ||
|- valign="top" | |- valign="top" | ||
| '''Dates:''' | | '''Dates:''' | ||
| Completion at end of | | Completion at end of Q2 2011 | ||
|- valign="top" | |- valign="top" | ||
| '''Status:''' | | '''Status:''' | ||
| Line 17: | Line 17: | ||
|- valign="top" | |- valign="top" | ||
| '''Repository Location:''' | | '''Repository Location:''' | ||
| | | https://github.com/geoelectric/mozmill-ondemand | ||
|- valign="top" | |- valign="top" | ||
| '''Tracking Bug / Bug List:''' | | '''Tracking Bug / Bug List:''' | ||
Revision as of 01:11, 14 May 2011
Overview
| Lead: | Geo Mealer |
| Co-workers: | Henrik Skupin |
| Dates: | Completion at end of Q2 2011 |
| Status: | Streamline how update testing is performed on releases |
| Repository Location: | https://github.com/geoelectric/mozmill-ondemand |
| Tracking Bug / Bug List: | bug 657081, bug 628659 |
| Documentation: | wiki |
Project details
Overview/Goal
Update testing of Firefox releases is currently an involved task with many manual steps. There have been opportunities in the past to parallelize or delegate the work that we've had to decline because of the difficulty of explaining how to set up, run, and analyze the results of the tests.
Our initial goal is to streamline the process into a semiautomatic solution that is easily documentable and does not require much manual handholding from the tester.
The Process Before
- Build an update test plan. Updates are performed from a given source to the current release build. A test plan chooses source builds using the following attributes:
- Platform (win32, mac64, etc.)
- Version (3.6, 3.6.12, etc.)
- Locale (en-US, es, etc.)
- Download each source build manually from ftp and stage in a directory hierarchy in a disk share.
- Manually log into platform machines/VMs that can see the above disk share, and (from command line) execute the update testing scripts. This is done once for each platform.
- Look on each machine for the generated test logs that result from the update testing script.
The Process after Phase I
Phase I concentrates on removing most of the fully manual steps in favor of a semi-automatic solution.
- Build an update test plan as above.
- From the test plan, create a configuration file including the desired builds, platforms, and locales.
- Run a single script that reads the configuration file, automatically downloads the desired builds, and stages them in the disk share.
- Run a single script that uses ssh to log into each target platform machine and executes the existing update testing script.
- Look in brasstacks (our automation result reporting framework) for the update test results as they land.
We have already moved to putting the results in brasstacks. The balance of Phase I requires creation of two scripts:
- The staging script
- The execution script
The Staging Script
The staging script should take as input a configuration file containing a list of all desired platforms, builds, and locales.
It should also take an option which is the root of the staging file system.
The script will expand the configuration file's lists into a comprehensive list of desired builds, and download into the staging area.
The Execution Script
The execution script should take as input a configuration file containing a list of target execution platforms.
The script will iterate through the list, logging into each platform and running the current update testing script. The current script already knows how to look in the staging area.
Login details will be handled via public/private key setup rather than name/PW to avoid having to write a secure way to encapsulate login info.