QA/Automation/Projects/Endurance Tests

Overview

Lead:	Dave Hunt
Second:	Anthony Hughes
Co-workers:	Aaron Train Henrik Skupin Geo Mealer
Dates:	Kick-off: December 2010 Phase I: Q1/2011 Phase II: Q2/2011 (goal)
Status:	Phase II in progress
Discussion:	General Test Snippets
Repository Location:	Shared libraries & tests: http://hg.mozilla.org/qa/mozmill-tests/ Automation scripts: http://hg.mozilla.org/qa/mozmill-automation/ Dashboard: https://github.com/whimboo/mozmill-dashboard
Tracking Bug(s):	bug 629065
Documentation:	Documentation

Meetings

Bi-weekly, Endurance Tests Project Meetings

• 1-800-707-2533 or 650-903-0800
• PIN 369 (if using 1-800 number)
• Ext. 92 (if using 650 number)
• Conference Number 315#

Backchannel: #mozmill

• 01/10/2011
• 01/19/2011
• 02/15/2011
• 03/01/2011
• 03/15/2011
• 04/08/2011

Summary

We're exploring different ways and or abilities to utilize Mozmill and its testing capabilities to determine if Firefox can sustain the continuous expected load seen through simulated 'real' web usage. These tests will ultimately create a simple real world model that employs a high but normal load pattern of actions for the browser in order to determine any potential:

• Browser performance degradation
• Crashes

Initially, this project will focus on setting up an environment employing a new test harness that will loop through simple test snippets to expose the above areas of concern within Firefox.

Definitions

Test Snippet

A test snippet will be one or more actions that can be performed within the AUT written in JavaScript according to the Mozmill specifications.

Roadmap

Phase I - Completed

1. Command line wrapper for Mozmill that:
   • Takes the following parameters:
     • Number of times to loop
     • Delay to wait before each iteration
   • Runs a single 'snippet' according to the above parameters
   • Outputs memory usage to the report
2. A 'snippet' that uses the new command line wrapper and outputs results that would potentially expose degradation of performance
3. New reports for endurance tests in Mozmill dashboard

Phase II (Draft)

1. Determine achievable additional performance metrics
2. Determine achievable client hardware details
3. Restart between each test by default (override from command line)
4. All tests must use local content
5. Improve performance of charts with considerable data
6. Create a view for results listed by memory consumption
7. Cache metrics to disk instead of storing in memory during the testrun

Phase ?

• Gracefully handle the AUT crashing, and ensure all reports are intact
• Output results to a location on the file system
• Read parameters in from disk
• Determine areas of perceived risk within the AUT
• Write more test snippets based on above
• Gather metrics on more system resources
• Present results in an easy to digest way
• Automate the running of endurance tests
• Abort tests that exceed predetermined resource thresholds
• Determining saturation points (baselines) in the AUT and monitor for major discrepancies

Additional Requirements

• Use debug nightly builds of Firefox to be able to get a full stack of information