Mobile/Testing/12 04 13

Previous Action Items

• snorp to try again to repro canvasmark failures on panda
  • see http://puppetagain.pub.build.mozilla.org/data/bmm/artifacts/panda-android-4.0.4_v3.1/ and https://bugzilla.mozilla.org/show_bug.cgi?id=914472#c3
• rel-eng to test sut 1.20 in staging
  • No problems discovered, plan to deploy live today.
• callek to disable canvasmark on pandas -- bug 916258
• jmaher to follow-up with blassey on talos -- bug 941155

Status reports

Rel Eng

• graydon has a spot on Ash to mess with Android 2.3 testing on EC2
  • https://tbpl.mozilla.org/?tree=Ash&jobname=Android%202.3
• Callek to deploy SUTAgent 1.20 today
• xpcshell on panda was not running on all trunk-based trees and did not ride the train, being fixed in bug 909457
• Blog post about Android Stats put up by Callek (see round table)

A Team

• tegra failure rate: [1.22%]
• panda failure rate: [13.6%]
  • excluding retries: 2.03

• Working on bug 937637 to get cpp unittests scheduled on trunk trees
• Can't attend this week due to a conflict

x86 automation

• All tests running hidden on Cedar, with some failures
• Complete:
  • kvm enabled and deployed: bug 929048
  • chunks coalesced, sets coalesced: bug 935214
  • patched emulator deployed: bug 933918
  • S1 split into 2 sets and output no longer duplicated: bug 939823
  • new AVD definitions (Graydon's) deployed: bug 919812
• Working on:
  • cleaning up remaining failures: bug 936226

Autophone

• Temporarily disabled Mochitest-Canvas due to instability in some phones. Will enhance Autophone to be able to run different tests on different devices and reenable this week.
• Nexus Devices
  • Getting 2 used Nexus 4. Try to locate 2012 Nexus 7 internally. When I have experience with the used devices will order new Nexus 5 and Nexus 7.
• Improving Autophone to help in detecting regression ranges.
  • Fixed bug 943871 - Autophone - use http instead of ftp for build urls
  • Fixed bug 943873 - Autophone - support inbound archive
    • Now able to use inbound archive of tinderbox builds for builds back to Mid May 2013.
  • Filed bug 945390 - Autophone - support build selection via revision
  • Planning to upgrade flot to 0.8.2 and improve graphs and date display on phonedash.mozilla.org
• Bugs
  • Found regression range for bug 916099 - Regression in "time to throbber stop" on Aug 28 2013
  • Plan to resolve INCOMPLETE bug 819984 - Regression in "time to throbber start" - Dec 5 and bug 825612 - Regression in "time to throbber start" and "time to throbber stop" - Dec 22 since they are outside inbound archive range and since we have other more important regressions to investigate.
  • Plan to file unfiled regressions with tight changesets from inbound archive. Removed obvious outliers from data and will file more recent regressions first.
  • Will also attempt to identify changesets which improve performance and comment in the associated bugs.

Round Table

• (armenzg) Re-thinking our Mobile testing
  • see http://blog.drapostles.org/archives/107 for info

We have awful wait times on Tegras

• We currently test Android 2.2 on Tegras
  • However, only less than 2% of our users are using Android 2.2
  • graydon is helping us to have an Android 2.3 story (10.82%-14.22% of our users)
• We currently test Armv6 builds (~2% of users) and no-ion (we don't ship this - only b2g18 branches)
  • I want to know if we can have a not-per-checkin model testing for Armv6
  • This would improve our current 2.2 testing on Tegras

I think we're OK with Pandas wrt to capacity, however, I want to point out that 4.1 has passed 4.0 in number of users.

Action Items

• snorp to try again to repro canvasmark failures on panda
  • see http://puppetagain.pub.build.mozilla.org/data/bmm/artifacts/panda-android-4.0.4_v3.1/ and https://bugzilla.mozilla.org/show_bug.cgi?id=914472#c3
• callek to disable canvasmark on pandas -- bug 916258
• jmaher to follow-up with blassey on talos -- bug 941155
• armenzg to send email about tegra retirement schedule, plans for armv6, android version coverage