Auto-tools/Goals/BigDreams

Project	Purpose	WAG time
Test Harnesses into Own Repo - (Independence Day)	Enables us to run tests on all versions without backporting harness infrastrucutre Makes developers able to write tests for any branch in any test type Easier to maintain Easier to catch breakage	1.5 - 2 Quarters
Universal Manifest - (Rosetta)	Makes all tests use the same manifest structure Gives us ability to manifest for mochi* tests Gives us the ability to handle different behaviors per product, per platform Gives us the ability to create a better manifest than what we have now Gives us flexibility to selectively enable/disable tests (could create an "orange" only run for debugging)	1 Quarter design and experiment with manifest, selling the idea to dev, refactoring mochitest/xpcshell to use a manifest at all. 1 month implementation reftest style harnesses 1 month porting and testing TODO: Would we want to keep old behavior going forward or just new behavior going forward? How does that change if we do the "test harness in own repo"?
Universal Test Runner - (Loderunner)	One test runner for all test types Simplifies development? Simplifies harness code? What problem does this solve? Lowers barrier to entry?	1 Quarter, design and implementation, refactoring wrapper for all test classes.
User Configuration/Dirty Profile Test Infrastructure - (Dirty Harry)	Re-use existing infrastructure for crash & valgrind infrastructure so that we can specify any type of profile we want. Create large bookmark/tag/history tests Vary extensions Vary saved passwords for sites Do this to test: startup time performance, page load performance, how our unit tests run in this configuration etc DEPENDS ON RESULT DB	1 month - experiment with extending sisyphus/infra to specify profile details 1 month to implement 1 month to craft specific perf tests
Universal Results Reporting Db - (Bookkeeper)	Allows us to store all results we need in one place Easy, extensible infrastructure to extend views to new data over time Open up an API to any dev member who crafts some kind of test to store data into the database Provide analytics using off-the-shelf tools to analyze results metrics (intermittent tests for example)	1 month design db, perform performance experiments 1 month implementation and begin on UI for views 1 month to complete UI for views, add in the APIs needed for ease of adding data into the db. 1-2 months reviews, unforseen complications
Litmus v2.0 - (Litmus 2.0)	Placeholder in case the Litmus project comes back to us ??	??
Mozmill 2.0	What do we need to get to 2.0? Will we do a 1.5 before that? (I think yes)	??
Coherent effort to find/fix [orange] issues - (Orangejuice)	Provide seamless integration for record/replay vms (already done?) Start busting a bunch of oranges	To make a dent we need to do at least 50 oranges before we see a noticeable decline in the rate of intermittent problems
Android Test Infra - (R2D2)	Find and fix issues with infrastructure on android Help RelEng with issues as they move ahead with putting infrastructure in staging ??	??
Crash/Assertion/Valgrind UI 2.0 - (CrystalBall)	The true potential of the crash/automation/valgrind information we have is to help us define and track down hard to reproduce crashes and to identify unstable pieces of code. The proper UI for this will demonstrate these correlations and allow dev's to view the relationships between this data. This is actually a formalization/aid of a process we do ad-hoc in bugzilla.	1 month, designs/prototypes 1-1.5 month implementation
Bugzilla Improvements - (Hydra)	We have talked a lot about improvements to bugzilla but we have never concretely listed what they are. What are they? What is actually desired/needed?	??
Symmetric Automation - (Doppelganger)	Creating a secure mirror repo and infrastructure for security fixes Not sure how much of this we need to do	Depends on how much of this is ours versus other groups
Crowd Sourced Unittests - (FeedGrass)	Allow for unittests to be run on various hardware models of beta testers. Sort of like GrafxBot, but for all tests. Ideal for Fennec where automation requires more resources and time while the hardware list is much larger.	Could piggyback off mozmill or grafxbot, need a lot of dev support. Probably 1 quarter to develop and get off the ground and 1 more quarter to polish up and evangalize
Toolset to engage users - (Powerball)	Addon in the browser to help detect user actions to broken functionality (web site wrong, plugin crash, bug in firefox, usability issue), then redirect to bugzilla Bugzilla interface to help file bug in appropriate place and search for dups Engage user after filing bug to reproduce or verify similar bugs on same hardware, version - better for triage Allow end user to earn points and share on social networks for filing, reproducing and verifying bugs Make this tangible to really build hype and competition between friends QMO 2011 Roadmap Provide ways to tell users when their bug is fixed that it is released in which version...put their names in the credits and they share with all their friends!	A lot of pieces here that are not even half baked. Just some ideas of a work flow that makes a lot of sense. This would be more of an ongoing project that could be done bits and pieces each quarter, or probably 1 quarter closer to full time.