Firefox OS/Performance/Power/Testing

Goals

The goals of the initial power consumption testing are as follows:

• Baseline version 1.1 power consumption performance for B2G on Hamatchi, Nexus S, and Nexus 4 hardware.
  • Detect power consumption regressions in 1.2 and later versions of B2G.
• Baseline Android ICS and JB on the Nexus S and Nexus 4 hardware.
  • Identify areas where B2G has non-trivially worse power consumption performance than Android.
• Gain the capability to triage, verify, and confirm leo+ bugs coming from hardware partners.

Methods

We are taking a black-box approach to power consumption testing. We are using a digital ammeter wired between the battery and the device to measure power consumption directly during testing runs. The problem of time correlation is to be solved initially by measuring the sampling delay in the COTS digital ammeter that we currently are using to measure current. Once I we know the mean time to first sample (MTFS) from the ammeter, then it is just a matter of coordinating the system clocks between the host PC and the handset device using NTP. The device will record timestamps for events in the test pass (e.g. simulated taps, app process start, etc) which will later be correlated with the samples coming from the ammeter which will be timestamped by the host PC.

We are building a custom ammeter and control harness for our in-house testing lab that will have firmware that supports time synchronization with the host PC, and reporting back of the time of first sample which will initialize the time correlation of the event data from the phone. Because the time stamp of the first sample will come from the time-coordinated ammeter, it will have higher corelation accuracy than the method used with the COTS ammeter.