canmove, Confirmed users
2,041
edits
B2G/QA/2014-10-31 Performance Acceptance: Difference between revisions
B2G/QA/2014-10-31 Performance Acceptance (view source)
Revision as of 21:16, 4 November 2014
, 4 November 2014→Video
(→Video) |
|||
| (22 intermediate revisions by the same user not shown) | |||
| Line 93: | Line 93: | ||
* p95: 1285 ms | * p95: 1285 ms | ||
'''Result''': ''' | '''Result''': '''FAIL''' (small regression, over guidelines) | ||
'''Comment''': | '''Comment''': Results are largely unchanged since 10-20. They remain slightly worse than 2.0 and over the 1000 ms target. | ||
==== Camera ==== | ==== Camera ==== | ||
| Line 114: | Line 114: | ||
* p95: 1668 ms | * p95: 1668 ms | ||
'''Result''': ''' | '''Result''': '''FAIL''' (over guidelines) | ||
'''Comment''': | '''Comment''': 2.1 results are slightly slower than the 10-20 comparison (~30 ms), but are close enough to possibly be within margin of error. As mentioned previously, though on paper the 2.1 results are slower than 2.0, the timing is actually different between branches and 2.1 is actually slightly faster. | ||
==== Clock ==== | ==== Clock ==== | ||
| Line 126: | Line 126: | ||
'''2.0''' | '''2.0''' | ||
* data points | * 390 data points | ||
* Median: | * Median: 901 ms | ||
* p95: | * p95: 1149 ms | ||
'''2.1''' | '''2.1''' | ||
* data points | * 450 data points | ||
* Median: | * Median: 972 ms | ||
* p95: | * p95: 1101 ms | ||
'''Result''': ''' | '''Result''': '''PASS''' | ||
'''Comment''': | '''Comment''': 2.1 results are slightly faster (~25 ms) than in the last comparison, but any improvement is arguably within margin of error. | ||
==== Contacts ==== | ==== Contacts ==== | ||
| Line 147: | Line 147: | ||
'''2.0''' | '''2.0''' | ||
* data points | * 390 data points | ||
* Median: | * Median: 743 ms | ||
* p95: | * p95: 856 ms | ||
'''2.1''' | '''2.1''' | ||
* data points | * 480 data points | ||
* Median: | * Median: 850 ms | ||
* p95: | * p95: 978 ms | ||
'''Result''': ''' | '''Result''': '''PASS''' | ||
'''Comment''': | '''Comment''': Results are fundamentally unchanged from the last comparison. | ||
==== Cost Control ==== | ==== Cost Control ==== | ||
[[File:2014-10- | [[File:2014-10-31 perf costcontrol.png|framed|left|FxOS Performance Comparison Results, 2.1 2014-10-31 Cost Control]] | ||
<br style="clear:both;" /> | <br style="clear:both;" /> | ||
'''2.0''' | '''2.0''' | ||
* data points | * 360 data points | ||
* Median: | * Median: 1609 ms | ||
* p95: | * p95: 1839 ms | ||
'''2.1''' | '''2.1''' | ||
* data points | * 480 data points | ||
* Median: | * Median: 2482 ms | ||
* p95: | * p95: 2607 ms | ||
'''Result''': ''' | '''Result''': '''FAIL''' (over guidelines, large regression) | ||
'''Comment''': | '''Comment''': Cost Control debuts in this comparison with times well over the target 1000 ms and with a large regression from 2.0. Investigation in {{bug|1072621}} has shown that this test only measures the first time the Usage app is run, where it runs through its own First Time Use flow; it was determined there that subsequent cold launches are significantly faster. | ||
==== Dialer ==== | ==== Dialer ==== | ||
| Line 187: | Line 187: | ||
'''2.0''' | '''2.0''' | ||
* | * 390 data points | ||
* Median: | * Median: 469 ms | ||
* p95: | * p95: 590 ms | ||
'''2.1''' | '''2.1''' | ||
* | * 450 data points | ||
* Median: | * Median: 539 ms | ||
* p95: | * p95: 613 ms | ||
'''Result''': ''' | '''Result''': '''PASS''' | ||
'''Comment''': | '''Comment''': Results are fundamentally the same as in the last comparison. | ||
==== FM Radio ==== | ==== FM Radio ==== | ||
| Line 208: | Line 208: | ||
'''2.0''' | '''2.0''' | ||
* data points | * 390 data points | ||
* Median: | * Median: 461 ms | ||
* p95: | * p95: 719 ms | ||
'''2.1''' | '''2.1''' | ||
* data points | * 450 data points | ||
* Median: | * Median: 490 ms | ||
* p95: | * p95: 698 ms | ||
'''Result''': ''' | '''Result''': '''PASS''' | ||
'''Comment''': | '''Comment''': 2.1 results are slightly improved from the last comparison, but possibly within margin of error. | ||
==== Gallery ==== | ==== Gallery ==== | ||
| Line 229: | Line 229: | ||
'''2.0''' | '''2.0''' | ||
* data points | * 390 data points | ||
* Median: | * Median: 873 ms | ||
* p95: | * p95: 1091 ms | ||
'''2.1''' | '''2.1''' | ||
* data points | * 450 data points | ||
* Median: | * Median: 954 ms | ||
* p95: | * p95: 1057 ms | ||
'''Result''': ''' | '''Result''': '''PASS''' | ||
'''Comment''': | '''Comment''': Results are fundamentally identical to the last comparison. | ||
==== Music ==== | ==== Music ==== | ||
| Line 250: | Line 250: | ||
'''2.1''' | '''2.1''' | ||
* data points | * 480 data points | ||
* Median: | * Median: 882 ms | ||
* p95: | * p95: 981 ms | ||
'''Result''': ''' | '''Result''': '''PASS''' | ||
'''Comment''': | '''Comment''': Music shows a slight slide from the last comparison (~35 ms). However, this is small enough to possibly be within margin of error. | ||
==== Settings ==== | ==== Settings ==== | ||
| Line 266: | Line 266: | ||
'''2.0''' | '''2.0''' | ||
* data points | * 390 data points | ||
* Median: | * Median: 3383 ms | ||
* p95: | * p95: 3734 ms | ||
'''2.1''' | '''2.1''' | ||
* | * 420 data points | ||
* Median: | * Median: 2823 ms | ||
* p95: | * p95: 3080 ms | ||
'''Result''': ''' | '''Result''': '''FAIL''' (over guidelines) | ||
'''Comment''': | '''Comment''': Settings has improved significantly from its previous slow point of 3131 ms on 10-20. It is not, however, back to the 2577 ms of the 10-02 comparison. The 2.0 results remain bimodal, and are unchanged even with fixed timing code applied to the branch. | ||
==== SMS ==== | ==== SMS ==== | ||
| Line 287: | Line 287: | ||
'''2.0''' | '''2.0''' | ||
* data points | * 390 data points | ||
* Median: | * Median: 1102 ms | ||
* p95: | * p95: 1286 ms | ||
'''2.1''' | '''2.1''' | ||
* data points | * 420 data points | ||
* Median: | * Median: 1206 ms | ||
* p95: | * p95: 1279 ms | ||
'''Result''': ''' | '''Result''': '''FAIL''' (over guidelines, regression) | ||
'''Comment''': | '''Comment''': The 2.1 results for SMS are very similar to the last comparison, within margin of error. The improved timing code on 2.0, however, makes it clear that there is a regression from that branch (measured at ~100 ms). | ||
==== Video ==== | ==== Video ==== | ||
| Line 308: | Line 308: | ||
'''2.0''' | '''2.0''' | ||
* data points | * 360 data points | ||
* Median: | * Median: 921 ms | ||
* p95: | * p95: 1136 ms | ||
'''2.1''' | '''2.1''' | ||
* data points | * 480 data points | ||
* Median: | * Median: 942 ms | ||
* p95: | * p95: 1045 ms | ||
'''Result''': ''' | '''Result''': '''PASS''' | ||
'''Comment''': | '''Comment''': Results are fundamentally the same as in the last comparison. | ||
== Raw Data == | == Raw Data == | ||
Will be added after full dataset is updated. | Will be added after full dataset is updated. | ||