View source for Auto-tools/Goals/BigDreams

{| class="standard-table"
|-
|class="header"|Project
|class="header"|Purpose
|class="header"|WAG time
|-
| '''Test Harnesses into Own Repo - (Independence Day)'''
|
* Enables us to run tests on all versions without backporting harness infrastrucutre
* Makes developers able to write tests for any branch in any test type
* Easier to maintain
* Easier to catch breakage
| 1.5 - 2 Quarters
|-
| '''Universal Manifest - (Rosetta)'''
|
* Makes all tests use the same manifest structure
* Gives us ability to manifest for mochi* tests
* Gives us the ability to handle different behaviors per product, per platform
* Gives us the ability to create a better manifest than what we have now
* Gives us flexibility to selectively enable/disable tests (could create an "orange" only run for debugging)
|
* 1 Quarter design and experiment with manifest, selling the idea to dev, refactoring mochitest/xpcshell to use a manifest at all.
* 1 month implementation reftest style harnesses
* 1 month porting and testing
* TODO: Would we want to keep old behavior going forward or just new behavior going forward?  How does that change if we do the "test harness in own repo"?
|-
| '''Universal Test Runner - (Loderunner)'''
|
* One test runner for all test types
* Simplifies development?  Simplifies harness code?
* What problem does this solve?
* Lowers barrier to entry?
| 1 Quarter, design and implementation, refactoring wrapper for all test classes.
|-
| '''User Configuration/Dirty Profile Test Infrastructure - (Dirty Harry)'''
|
* Re-use existing infrastructure for crash & valgrind infrastructure so that we can specify any type of profile we want.
* Create large bookmark/tag/history tests
* Vary extensions
* Vary saved passwords for sites
* Do this to test: startup time performance, page load performance, how our unit tests run in this configuration etc
* <font color="red">DEPENDS ON RESULT DB</font>
| 
* 1 month - experiment with extending sisyphus/infra to specify profile details
* 1 month to implement
* 1 month to craft specific perf tests
|-
| '''Universal Results Reporting Db - (Bookkeeper)'''
|
* Allows us to store all results we need in one place
* Easy, extensible infrastructure to extend views to new data over time
* Open up an API to any dev member who crafts some kind of test to store data into the database
* Provide analytics using off-the-shelf tools to analyze results metrics (intermittent tests for example)
|
* 1 month design db, perform performance experiments
* 1 month implementation and begin on UI for views
* 1 month to complete UI for views, add in the APIs needed for ease of adding data into the db.
* 1-2 months reviews, unforseen complications
|-
| '''Litmus v2.0 - (Litmus 2.0)'''
|
* Placeholder in case the Litmus project comes back to us
* ??
|??
|-
| '''Mozmill 2.0'''
|
* What do we need to get to 2.0?
* Will we do a 1.5 before that? (I think yes)
|
??
|-
| '''Coherent effort to find/fix [orange] issues - (Orangejuice)'''
|
* Provide seamless integration for record/replay vms (already done?)
* Start busting a bunch of oranges
| 
To make a dent we need to do at least 50 oranges before we see a noticeable decline in the rate of intermittent problems
|-
| '''Android Test Infra - (R2D2)'''
|
* Find and fix issues with infrastructure on android
* Help RelEng with issues as they move ahead with putting infrastructure in staging
* ??
|?? 
|-
| '''Crash/Assertion/Valgrind UI 2.0 - (CrystalBall)'''
|
* The true potential of the crash/automation/valgrind information we have is to help us define and track down hard to reproduce crashes and to identify unstable pieces of code.
* The proper UI for this will demonstrate these correlations and allow dev's to view the relationships between this data.
* This is actually a formalization/aid of a process we do ad-hoc in bugzilla.
|
* 1 month, designs/prototypes
* 1-1.5 month implementation
|-
| '''Bugzilla Improvements - (Hydra)'''
|
* We have talked a lot about improvements to bugzilla but we have never concretely listed what they are.
* What are they?
* What is actually desired/needed?
|
??
|-
| '''Symmetric Automation - (Doppelganger)
|
* Creating a secure mirror repo and infrastructure for security fixes
* Not sure how much of this we need to do
| Depends on how much of this is ours versus other groups
|-
| '''Crowd Sourced Unittests - (FeedGrass)
|
* Allow for unittests to be run on various hardware models of beta testers.  Sort of like GrafxBot, but for all tests.
* Ideal for Fennec where automation requires more resources and time while the hardware list is much larger.
| Could piggyback off mozmill or grafxbot, need a lot of dev support.
|}