QA/TDAI/2009 Q4 Sprints

We have three sprints in Q4 as you can easily see from https://wiki.mozilla.org/QA/TDAI/Goals/2009-Q4.

Sprint 1 Review: October 23

• Mobile Test Framework
  • Quick Demo
  • Where we Are
    • Basic DeviceAgent with 90% functionality implemented
    • Working scripts for mochitest, xpcshell, and talos to utilize DeviceAgent
  • Next Steps
    • Heartbeat and DeviceInfo from Device->TestAgent
    • Integrate harness scripts into TestAgent, including download/install, verify, mochitest, xpcshell
    • Upload logs, status, and updates to TestServer

• Valgrind Test Automation
  • (no demo -- too slow!)
    • show examples of valgrind log output
  • Where we are
    • We have support for running valgrind with jsreftests, crashtests, reftests, xpcshell tests, and mochitests. All relevant patches have landed: bug 521130, bug 521457
  • Next Steps
    • Collect valgrind exclusion list, for excluding certain uninteresting errors.
    • Investigate CPU/RAM requirements for VM's.
    • Investigate use of valgrind 3.2.1 (ships with CentOS 5) vs 3.5.
    • Investigate log parsing: XML output, error summaries.

• Automatic Crash Reporting
  • Demo?
  • Where we are
  • Next Steps
    • How close are we to turning it on?

• Electrolysis Testing
  • Demo?
  • Where we are
    • State of OOP Plugin Tests
    • State of OOP Content Tests
  • Next Steps
    • OOP Plugins Tests
    • OOP Plugins Talos
    • OOP Content Tests
    • OOP Content Talos

• XBL 2
  • Demo
  • Where we are
  • Next Steps (beyond Q4)

• Jetpack Test Harness
  • Demo
  • Where we are
  • Next Steps

• Log Compare
  • (no demo)
  • Where we are
    • Fennec?
    • Firefox?
  • Next Steps
    • Fennec
    • Firefox

• Adobe Test Harness Automation
  • (no demo)
  • Where we are
    • Waiting on Adobe for links/access
  • Next Steps
    • Jgriffin will own getting it running and automated if we ever hear back from Adobe

• HTML 5 W3C Tests
  • (no demo)
  • Where We Are
    • Waiting on W3C to start doing stuff
  • Next Steps

Things Needing Attention

• Gfx test harness
  • URGENT:: Need: Mechanism to measure "choppiness" of video for full screen tests
  • Need: Mechanism to test hardware accelerated 3d via extension
  • Need: Research into types of tests to write
  • Need: Research into WebGL implementation

• E10S Talos integration
  • Can we get help from RelEng?
  • Does Jmaher have to own?

• JS Ref Test
  • Browser Tests: Can we get this on Tinderbox quicker?
  • Shell Tests: Can we get these completed quicker?