B2G/QA/2014-10-31 Performance Acceptance
2014-10-31 Performance Acceptance Results
Overview
These are the results of performance release acceptance testing for FxOS 2.1, as of the Oct 31, 2014 build.
Our acceptance metric is startup time from launch to visually-complete, as metered via the Gaia Performance Tests, with the system initialized to make reference-workload-light.
For this release, there are two baselines being compared to: 2.0 performance and our responsiveness guidelines targeting no more than 1000ms for startup time.
The Gecko and Gaia revisions of the builds being compared are:
2.0:
- Gecko: mozilla-b2g32_v2_0/82a6ed695964
- Gaia: 7b8df9941700c1f6d6d51ff464f0c8ae32008cd2
2.1:
- Gecko: mozilla-b2g34_v2_1/50d48f9a04c7
- Gaia: f89c7b12c36572262c9ea76058694a139b1a8634
Note when comparing to past reports that the test timings changed with Bug 1079700 and a handful of subsequent followups. The last report also noted this, as it affected the 2.1 timings. At the time, it was thought that the 2.0 timings were a control, but we found that the patch actually did not fix 2.0. All fixes on both branches are now in place, hence faster 2.0 timings than previously noted.
This initial report has a limited (390 maximum) data point set for 2.0. It will be updated with full results for future baselines and any notable differences (unlikely) will be communicated.
Startup -> Visually Complete
Startup -> Visually Complete times the interval from launch when the application is not already loaded in memory (cold launch) until the application has initialized all initial onscreen content. Data might still be loading in the background, but only minor UI elements related to this background load such as proportional scroll bar thumbs may be changing at this time.
This is equivalent to Above the Fold in web development terms.
More information about this timing can be found on MDN.
Execution
These results were generated from 480 application data points per release, generated over 16 different runs of make test-perf as follows:
- Flash to base build
- Flash stable FxOS build from tinderbox
- Constrain phone to 319MB via bootloader
- Clone gaia
- Check out the gaia revision referenced in the build's sources.xml
- GAIA_OPTIMIZE=1 NOFTU=1 make reset-gaia
- make reference-workload-light
- For up to 16 repetitions:
- Reboot the phone
- Wait for the phone to appear to adb, and an additional 30 seconds for it to settle.
- Run make test-perf with 31 replicates
Result Analysis
First, any repetitions showing app errors are thrown out.
Then, the first data point is eliminated from each repetition, as it has been shown to be a consistent outlier likely due to being the first launch after reboot. The balance of the results are typically consistent within a repetition, leaving 30 data points per repetition.
These are combined into a large data point set. Each set has been graphed as a 32-bin histogram so that its distribution is apparent, with comparable sets from 2.0 and 2.1 plotted on the same graph.
For each set, the median and the 95th percentile results have been calculated. These are real-world significant as follows:
- Median
- 50% of launches are faster than this. This can be considered typical performance, but it's important to note that 50% of launches are slower than this, and they could be much slower. The shape of the distribution is important.
- 95th Percentile (p95)
- 95% of launches are faster than this. This is a more quality-oriented statistic commonly used for page load and other task-time measurements. It is not dependent on the shape of the distribution and better represents a performance guarantee.
Distributions for launch times are positive-skewed asymmetric, rather than normal. This is typical of load-time and other task-time tests where a hard lower-bound to completion time applies. Therefore, other statistics that apply to normal distributions such as mean, standard deviation, confidence intervals, etc., are potentially misleading and are not reported here. They are available in the summary data sets, but their validity is questionable.
On each graph, the solid line represents median and the broken line represents p95.
Pass/Fail Criteria
Pass/Fail is determined according to our documented release criteria for 2.1. This boils down to launch time being under 1000 ms.
Median launch time has been used for this, per current convention. However, as mentioned above, p95 launch time might better capture a guaranteed level of quality for the user. In cases where this is significantly over 1000 ms, more investigation might be warranted.
Results
Calendar
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Camera
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Clock
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Contacts
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Cost Control
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Dialer
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
FM Radio
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment:
Gallery
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Music
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Settings
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
SMS
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Video
2.0
- data points
- Median:
- p95:
2.1
- data points
- Median:
- p95:
Result: TBD
Comment: TBD
Raw Data
Will be added after full dataset is updated.