Auto-tools/Goals/2013Q4

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

Top Level

Highest priority, multi-team dependent goals

Goal	Measurement	Status	Owner	Bug	Area
Run endurance tests without Marionette issues	Run a set of endurance tests without gaia actions (or with simplified gaia actions to test only marionette) for 3 days w/o issues	[ON TRACK]	Mdas	TBD	Marionette
Deploy new B2G desktop tests, reftest running OOP, and gaia integration to TBPL	All these suites running on B2G Inbound	[ON TRACK]	Jgriffin	bug 916356	B2G
Support B2G Perf Team with automated power testing and Gaia Productivity team with github autoland	Provide aid, expertise, and reviews to ensure teams are successful	[ON TRACK]	Wlach	TBD	B2G
Deploy automated B2G Eideticker system to run nightly	Successful completion on deadline	[ON TRACK]	Wlach	TBD	B2G
Create and Deploy one dashboard for docs, usage statistics, and tools for all our performance tools	Completion of the project by deadline	[ON TRACK]	Jmaher	TBD	Performance
Implement and deploy a fully functional alerting system for performance regressions and data ingestion monitoring.	Completion of project by end of quarter	[ON TRACK]	Jeads	TBD	Datazilla
Ensure sheriffs can perform side by side usability testing in TBPL	Complete starring, orange factor integration, plugin detail pane, (possibly) regex to tag transformation by end of quarter	[ON TRACK]	Mdoglio	TBD	Treeherder
Continue Bugzilla as a Platform transition	Deploy BzAPI compatibility layer, Change notification POC, Memcache minimum viable product by end of quarter	[ON TRACK]	Glob	TBD	Bugzilla
Force Android reftests to run within 15% of their original, pre skia time	Ensure panda and x86 reftests run within 15% of their original time	[ON TRACK]	Gbrown	TBD	Android
Ensure steeplechase webRTC automation as hands off as possible	Deploy C++ tests, mochitests, ability to change machine configuration to system by end of quarter	[ON TRACK]	Ted	TBD	WebRTC
Deploy bugmaster community building tools: bug board, tutorial, and bugzilla user profile report	Tools deployed by end of quarter	[ON TRACK]	Lizzard	TBD	Bugmaster
Deploy mozmill 2.0, Metro QA tests, and aid softvision to take over day to day operations for QA automation with our oversight	Projects deployed, and 15 consecutive days were SV team works with minimal oversight	[ON TRACK]	Whimboo	TBD	QA Automation
Operationalize sheriffing role and complete documentation so that we can pull in community sheriffs	Have documentation completed, identify one part time community sheriff.	[ON TRACK]	Edmorley	TBD	Sheriffing

Detailed Lists

These contain the top level goals as well as the supporting, ongoing, and additional projects we'll try to complete this quarter. Note that additional projects may be moved to the backlog to cope with changes in priority or new projects. The current backlog is here

Marionette (mdas)

• P0 - Run set of endurance tests for 3 days without marionette issues
  • space issues, memory usage, import log, and don't regress socket.timeout

Ongoing and Supporting Projects
[ON TRACK] P0 - (crossteam)Get passing WebDriver conformance tests running in TBPL [ON TRACK] P0 - (crossteam)Fix touch related bugs [ON TRACK] P0 - (ongoing) Continue to support WebQA and Gaia teams with bug fixes and new features

B2G (jgriffin)

• P0 [ahal] (highlight) mochitests, reftests, crashtests, jsreftests, gaia-ui-tests on b2g desktop builds: debug/opt linux32/64, debug/opt macosx64
• P0 [ahal] reftests/crashtests running out of process
• P0 [jgriffin] gaia integration tests on tbpl
• P0 [wlach supporting] (crossteam) (perf team) power consumption testing
• P0 [rwood] (crossteam) memory tests
• P0 [wlach,davehunt] Deploy automated Eideticker system to run nightly for Firefox OS
• P0 [wlach] Deliver a suite of Eideticker responsiveness tests for FirefoxOS
• P1 [wlach supporting] (crossteam)(with gaia team) github autolanding
• P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system

New Builds and Tests

• P1 [ahal] tests on 4.2 emulators (https://bugzilla.mozilla.org/show_bug.cgi?id=912249)
• P1 [ahal] jsreftests running out of process and enabled on all branches (and disable android noion)
• P1 [jgriffin, ahal] mochitests, reftests, crashtests, jsreftests, webapi, and xpcshell tests on debug emulator builds: linux64
• P1 [jgriffin, davehunt] get gaia-ui-tests going on emulators on TBPL
• P2 [jgriffin] run "sanity test" of emulator on osx

Ongoing and Supporting Projects
P1 [jgriffin, ahal] continued work to make B2G tests more sheriffable P2 [jgriffin, ahal] (internal) continued burn down of intermittent oranges P2 [ahal] continued work to make B2G tests more developer friendly

Performance (jmaher)

• Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard)

• P0 [jmaher] Complete investigation into Talos suites with desktop performance team
• P0 [jmaher, ekyle] Complete alerting analysis for datazilla data
• P0 [jmaher] hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority
• P1 [jmaher, bc] Investigate desktop power testing

Ongoing and supporting projects
have tools, data, tests fully documented and easy to find (all) proactively meet with at least 5 teams or team leaders to get work in a pipeline instead of an emergency. (jmaher) Ensure JS Team is satisfied with real world benchmarking system, investigate automation Build performance lab Continue supporting games benchmarking effort

Datazilla (jeads)

• The Q4 top priority is to implement a fully functional alerting system for performance regressions and data ingestion monitoring.

• Implement alert system that monitors data ingestion rates for all branches in all projects.
  • NOTE: Data ingestion rates should be reported in a graph, this could be part of the performance dashboard, abnormal values should trigger alert emails.
• Fine tune automated regression detection system as needed
• Implement the performance dashboard described in the performance section

Ongoing and supporting projects

Set up eideticker projects and backfill eideticker data (https://bugzilla.mozilla.org/show_bug.cgi?id=909796) Integrate documentation in the homepage UI and in the b2g UI Remove all legacy code from deprecated UI's \o/ NOTE: It's possible that slow page load performance in the new UI will require us to implement a reference data cache. Not sure if this will be an issue yet but something to be aware of. NOTE: There are several tasks building up in the b2g performance ui. They're assembled here, bug 913726, not sure how these should be prioritized in Q4.

Treeherder (mdoglio/jeads)

• Ensure sheriffs can perform side by side usability testing in TBPL

• Implement TBPL *starring* features
• Implement Orange Factor compatability
• Release production instance on AWS
• Make sure we're on target for full TBPL replacement in Q1 2014.

Bugzilla (glob/mcote)

• Q4 top priority is to continue to focus on "Bugzilla as a Platform". This is a multifaceted goal covering API design, documentation authoring, and community engagement, and of course development.

• Continue Bugzilla as a Platform (BaaP):
  • [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval)
  • [dkl] (crossteam) bzAPI compatibility layer
  • [dkl] plan for versioning APIs, first draft of interface for API v2
• [glob] (crossteam) memcached minimum viable product (also helps BaaP above)

Ongoing and supporting projects
[mcote] ReviewBoard integration plan based on feedback [mcote] somehow get over to git, or at least have a really serious plan for it [kyle] deploy new bzETL

Android (gbrown)

• Q4 top priority is remote reftest profiling in support of running reftests in reasonable time on Android 4.0/Panda and Android 4.2/x86.

• mach commands for running remote tests
  • automatically get xpcshell/xre for remote tests
  • [dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)?
• Eideticker responsiveness (possible carry over, depending on Q3).

Ongoing and supporting projects
Continue working with releng to increase panda stability [bc] Use Autophone to do one-off cross browser bandwidth metrics Deploy C++ remote tests

WebRTC (ted)

• Port all webRTC mochitests to SteepleChase
• Ensure SteepleChase runs on as close to every commit as possible
• Work with Releng to investigate Deployment to Releng Cloud, per change testing
• Deploy C++ tests for SteepleChase
• Deploy ability to change machine configuration per test
• Investigate hardware testing set up (joint with desktop QA)

Bugmaster (lizzard)

• Deploy bugmaster community building tools (bug board, tutorial, and bugzilla user profile report)
• Use summit to build out community involvement
• Continue triaging and imrpove the groundswell.
• Research into new Bugzilla home page for logged-in/logged-out users (separate pages)

Desktop QA Automation (whimboo)

• (crossteam) Get SoftVision to lead the automation efforts for QA with our oversight
• (crossteam) Switch Mozmill CI from Mozmill 1.5.x to Mozmill 2.0
• (crossteam) Get Firefox Metro tests executed in production

Ongoing and supporting projects
Extend Mozmill CI to cover most of our localized beta and release candidate builds Create web application to generate the beta and release Jenkins job configuations (stretched) Setup tryserver Mozmill CI system assist team members in testing their patches (stretched) Investigate requirements for community driving tool by working together with jdm, lizzard, and pieros

sheriffs (edmorley)

• Test treeherder throughout the quarter, file bugs
• Complete documentation goals/operationalize sheriffing
• Have 1 new community sheriff

mozbase (ahal)

• add structured logging to the mochitest harness
  • need to merge the logs from python and js
• determine a long term vision