Auto-tools/Goals/2013Q4

Top Level

Highest priority, multi-team dependent goals

Goal	Measurement	Status	Owner	Bug	Area
Run endurance tests without Marionette issues	Run a set of endurance tests without gaia actions (or with simplified gaia actions to test only marionette) for 3 days w/o issues	[DONE]	Mdas	bug 908261 has been marked a dupe, what will help here is bug 931044 which we could hook up to run for 3 days	Marionette
Deploy new B2G desktop tests, reftest running OOP, and gaia integration to TBPL	All these suites running on B2G Inbound	[DONE]	Jgriffin	bug 916356	B2G
Support B2G Perf Team with automated power testing and Gaia Productivity team with github autoland	The power testing goal was deferred due to lack of a suitable device, and the Gaia team didn't end up needing our help with the first iteration of github autoland.	[DEFER]	Wlach	bug 917375, Github autoland bot	B2G
Deploy automated B2G Eideticker system to run nightly	Successful completion on deadline	[DONE]	Wlach	bug 916989	B2G
Create and Deploy one dashboard for docs, usage statistics, and tools for all our performance tools	This goal's priority was downgraded to allow Joel to spend time on higher-priority perf items, including desktop power testing, xperf, and webpagetest.	[MISSED]	Jmaher	bug 919619	Performance
Implement and deploy a fully functional alerting system for performance regressions and data ingestion monitoring.	Jeads time was better spent on Treeherder; we will target this goal for Q12014.	[MISSED]	Jeads	bug 921218	Datazilla
Ensure sheriffs can perform side by side usability testing in TBPL	Complete starring, orange factor integration, plugin detail pane, (possibly) regex to tag transformation by end of quarter. This should be done, except for starring and orange factor integration.	[DONE]	Mdoglio	Pivotal Tracker Instance	Treeherder
Continue Bugzilla as a Platform transition	Deploy BzAPI compatibility layer, Change notification POC, Memcache minimum viable product by end of quarter	[DONE] (except bzAPI compatibility layer, eta early Q1)	Glob	bug 885464	Bugzilla
Force Android reftests to run within 15% of their original, pre skia time	Ensure panda and x86 reftests run within 15% of their original time	[DONE]	Gbrown	bug 916657	Android
Ensure steeplechase webRTC automation as hands off as possible	Deployed mochitests, ability to change machine configuration to system by end of quarter. C++ tests were not created (not our responsibility) so did not get integrated.	[DONE]	Ted	bug 894559	WebRTC
Deploy bugmaster community building tools: bug board, tutorial, and bugzilla user profile report	Tools deployed by end of quarter	[MISSED]	Lizzard	bug 919624	Bugmaster
Deploy Mozmill 2.0, Metro infrastructure and tests, and aid SoftVision to take over day to day operations for QA automation with our oversight	Deployed projects, tried to get the SV team to take over the reigns, and we had marginal success. Had to bail them out a few times.	[DONE]	Whimboo	bug 922200	QA Automation
Operationalize sheriffing role and complete documentation so that we can pull in community sheriffs	Have documentation completed, identify one part time community sheriff.	[DONE]	Edmorley	bug 928960	Sheriffing

Detailed Lists

These contain the top level goals as well as the supporting, ongoing, and additional projects we'll try to complete this quarter. Note that additional projects may be moved to the backlog to cope with changes in priority or new projects. The current backlog is here

Marionette (mdas)

• [DONE] P0 - Run set of endurance tests for 3 days without marionette issues
  • space issues, memory usage, import log, and don't regress socket.timeout

Ongoing and Supporting Projects
P0 - (crossteam)Get passing WebDriver conformance tests running in TBPL P0 - (crossteam)Fix touch related bugs P0 - (ongoing) Continue to support WebQA and Gaia teams with bug fixes and new features

B2G (jgriffin)

• [DONE] P0 [ahal] (highlight) mochitests, reftests, gaia-ui-tests on b2g desktop builds: debug/opt linux32/64, debug/opt macosx64
• [MISSED] P0 [ahal] crashtests, jsreftests on b2g desktop builds
• [DONE] P0 [ahal] reftests/crashtests running out of process
• [DONE] P0 [jgriffin] gaia integration tests on tbpl
• [DEFER] P0 [wlach supporting] (crossteam) (perf team) power consumption testing
• [MISSED] P0 [rwood] (crossteam) memory tests
• [DONE] P0 [wlach,davehunt] Deploy automated Eideticker system to run nightly for Firefox OS
• [MISSED] P0 [wlach] Deliver a suite of Eideticker responsiveness tests for FirefoxOS
• [DROPPED] P1 [wlach supporting] (crossteam)(with gaia team) github autolanding
• [DONE] P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system

New Builds and Tests

• [DEFER] P1 [ahal] tests on 4.2 emulators (https://bugzilla.mozilla.org/show_bug.cgi?id=912249)
• [MISSED] P1 [ahal] jsreftests running out of process and enabled on all branches (and disable android noion)
• [MISSED] P1 [jgriffin, ahal] mochitests, reftests, crashtests, jsreftests, webapi, and xpcshell tests on debug emulator builds: linux64
• [MISSED] P1 [jgriffin, davehunt] get gaia-ui-tests going on emulators on TBPL
• [DEFER] P2 [jgriffin] run "sanity test" of emulator on osx

Ongoing and Supporting Projects
P1 [jgriffin, ahal] continued work to make B2G tests more sheriffable P2 [jgriffin, ahal] (internal) continued burn down of intermittent oranges P2 [ahal] continued work to make B2G tests more developer friendly

Performance (jmaher)

• [MISSED] Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard)

• [DONE] P0 [jmaher] Complete investigation into Talos suites with desktop performance team
  • tsvg, tscroll, ts, tpaint are all refreshed/updated accordingly.
• [MISSED] P0 [jmaher, ekyle] Complete alerting analysis for datazilla data
• [DONE] P0 [jmaher] hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority
• [DONE] P1 [jmaher, bc] Investigate desktop power testing

Ongoing and supporting projects
have tools, data, tests fully documented and easy to find (all) proactively meet with at least 5 teams or team leaders to get work in a pipeline instead of an emergency. (jmaher) Ensure JS Team is satisfied with real world benchmarking system, investigate automation Build performance lab Continue supporting games benchmarking effort

Datazilla (jeads)

• [MISSED] The Q4 top priority is to implement a fully functional alerting system for performance regressions and data ingestion monitoring.

• Implement alert system that monitors data ingestion rates for all branches in all projects.
  • NOTE: Data ingestion rates should be reported in a graph, this could be part of the performance dashboard, abnormal values should trigger alert emails.
• Fine tune automated regression detection system as needed
• Implement the performance dashboard described in the performance section

Ongoing and supporting projects

Set up eideticker projects and backfill eideticker data (https://bugzilla.mozilla.org/show_bug.cgi?id=909796) Integrate documentation in the homepage UI and in the b2g UI Remove all legacy code from deprecated UI's \o/ NOTE: It's possible that slow page load performance in the new UI will require us to implement a reference data cache. Not sure if this will be an issue yet but something to be aware of. NOTE: There are several tasks building up in the b2g performance ui. They're assembled here, bug 913726, not sure how these should be prioritized in Q4.

Treeherder (mdoglio/jeads)

• [DONE] Ensure sheriffs can perform side by side usability testing in TBPL

• [MISSED] Implement TBPL *starring* features
• [MISSED] Implement Orange Factor compatability
• [MISSED] Release production instance on AWS
• [DONE] Make sure we're on target for full TBPL replacement in Q1 2014.

Bugzilla (glob/mcote)

• [DONE] Q4 top priority is to continue to focus on "Bugzilla as a Platform". This is a multifaceted goal covering API design, documentation authoring, and community engagement, and of course development.

• Continue Bugzilla as a Platform (BaaP):
  • [DONE] [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval)
  • [MISSED] [dkl] (crossteam) bzAPI compatibility layer
  • [MISSED] [dkl] plan for versioning APIs, first draft of interface for API v2
    • We had some interruptions due to more API work, developer tools, and antispam efforts.
• [DONE] [glob] (crossteam) memcached minimum viable product (also helps BaaP above)

Ongoing and supporting projects
[DEFER] [mcote] ReviewBoard integration plan based on feedback Discussion is still ongoing as to whether Review Board can meet our needs or not. [DONE] [mcote] somehow get over to git, or at least have a really serious plan for it Plan in place. [DONE] [kyle] deploy new bzETL

Android (gbrown)

• [DONE] Q4 top priority is remote reftest profiling in support of running reftests in reasonable time on Android 4.0/Panda and Android 4.2/x86.

• [MISSED] mach commands for running remote tests
  • automatically get xpcshell/xre for remote tests
  • [dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)?
• [DONE] Eideticker responsiveness (possible carry over, depending on Q3).

Ongoing and supporting projects
Continue working with releng to increase panda stability [bc] Use Autophone to do one-off cross browser bandwidth metrics [DONE] [dminor] Deploy C++ remote tests

WebRTC (ted)

• [DROPPED] P0: Get Steeplechase into releng staging (joint with releng)
• [MISSED] P0: Deploy C++ tests for SteepleChase
• [DONE] P0: Deploy ability to change machine configuration per test
• [DEFER] P1: Investigate hardware testing set up (joint with desktop QA)

Bugmaster (lizzard)

• [MISSED] Deploy bugmaster community building tools (bug board, tutorial, and bugzilla user profile report)
• [DONE] Use summit to build out community involvement
• [DONE] Continue triaging and imrpove the groundswell.
• [MISSED] Research into new Bugzilla home page for logged-in/logged-out users (separate pages)

Desktop QA Automation (whimboo)

• [DONE] (crossteam) Get SoftVision to lead the automation efforts for QA with our oversight
• [DONE] (crossteam) Switch Mozmill CI from Mozmill 1.5.x to Mozmill 2.0
• [MISSED] (crossteam) Get Firefox Metro tests executed in production

Ongoing and supporting projects
[MISSED] Extend Mozmill CI to cover most of our localized beta and release candidate builds [DONE] Create web application to generate the beta and release Jenkins job configuations (stretched) Setup tryserver Mozmill CI system assist team members in testing their patches (stretched) Investigate requirements for community driving tool by working together with jdm, lizzard, and pieros

sheriffs (edmorley)

Meta bug: bug 928960

• [DEFER] Test treeherder throughout the quarter, file bugs
• [DONE] Complete documentation goals/operationalize sheriffing
• [DONE] Have 1 new community sheriff

mozbase (ahal)

• [MISSED] add structured logging to the mochitest harness
  • need to merge the logs from python and js
• [DONE] determine a long term vision