Firefox OS/Performance/Automation/Launch Latency: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (Lakrits moved page FirefoxOS/Performance/Automation/Launch Latency to Firefox OS/Performance/Automation/Launch Latency: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the sp...)
 
(55 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Launch Latency =
= Launch Latency =
== Summary ==
== Summary ==


Measures elapsed time from the first launch of an application to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].
Measures elapsed time from the first launch of an application to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].


== Precision ==
== Usage ==
 
''This would be filled in with the estimated precision of the test. For example, we might trust the results within 10ms or we might trust them within 100ms. The latter would imply very different margins before we could assume a regression.''


== Scenarios ==
=== Test Cases ===


Launch each of the following apps from a cold state:
Launch each of the following apps from a cold state:


* Browser
* Browser
* Calendar
* Camera
* Camera
* Clock
* Contacts
* Contacts
* Dialer
* Dialer
* Email
* Email
* FM Radio
* Gallery
* Gallery
* Marketplace
* Marketplace
* Music
* Settings
* Settings
* SMS
* SMS
* Template
* Usage
* Video


For each app, report the time elapsed to the following events:
For each app, report the time elapsed to the following events:
Line 30: Line 36:
* moz-app-loaded
* moz-app-loaded


== Dashboards ==
=== Precision ===


''This should be a series of links to dashboards per branch, device, and test. Unfortunately, Datazilla doesn't seem to allow direct linking without embedding revisions, specific implemented apps, and other time-specific artifacts in the URL. This is a real problem.''
TBD


=== 2.0 ===
=== How to Run On-Demand ===
 
TBD
 
=== Published Results ===
 
==== FxOS 2.0 ====


{| class="wikitable"
{| class="wikitable"
! Test
!
! Flame
! Flame
|-
|-
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame&range=7&test=startup_%3E_moz-app-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
|}
|}


=== 2.1 ===
==== FxOS 2.1 ====


{| class="wikitable"
{| class="wikitable"
! Test
!
! Flame
! Flame
|-
|-
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-dom-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-dom-loaded&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-chrome-interactive''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-chrome-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-visually-complete''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-content-interative''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median link]
| '''startup_>_moz-app-loaded''' || [https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-content-interactive&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,video&app=calendar&gaia_rev=8cb1a949f2e9650b&gecko_rev=c6f9f0396153&plot=median Datazilla]
|-
|-
|}
|}


== Procedure ==
=== References ===


''This needs more detail.''
* [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines Firefox OS App Responsiveness Guidelines]


# Clear app from memory
== Development ==
# Inject launch command
# Monitor app events for each of the five events under Scenarios
# Time elapsed is the time between that event and ''start time?''


== References ==
=== Design ===


TBD
==== Common Setup ====
# Set up workload
# Restart B2G process
 
==== All Cases ====
# Invoke launch programmatically
# Inject the Performance Helper and save start time
# App instrumentation throws timeline events at appropriate times
# Performance helper observes events
# Event emit times are compared with start time for time elapsed


== Roadmap ==
==== Results ====
* Result is time elapsed to each event
* Execute test 30 times, for each event report mean, median, deviation


=== Overview ===
=== Project Roadmap ===


* '''Lead:''' Eli Perelman
* '''Lead:''' Eli Perelman
* '''Tracking Bug:''' TBD
* '''Tracking Bug:''' TBD
=== Milestone 1: Design ===
'''Tracking Bug:''' TBD


{| class="wikitable"
{| class="wikitable"
! width="120px" |  
! width="120px" |
! Target Release
! colspan="2" | Milestone 1: Design
! Test Design  
|-
!
! Test Design
! Validity Review
! Validity Review
|-
|-
| '''All''' || TBD || Bug || Bug
| '''All Cases''' || Bug || Bug
|-
|-
|}
|}
=== Milestone 2: Test ===
'''Tracking Bug:''' TBD


{| class="wikitable"
{| class="wikitable"
! width="120px" |
! width="120px" |
! Target Release
!
! colspan="4" | Milestone 2: Test
! colspan="2" | Milestone 3: Publication
|-
!
! Availability
! Instrumentation
! Instrumentation
! Workloads
! Workload
! Single-run test
! On-Demand Test
! Results Review
! Results Review
! Published Results
! Documentation
|-
|-
| '''Browser''' || TBD || Bug || Bug || Bug || Bug
| '''Browser''' || TBD || {{bug|837668}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Camera''' || TBD || Bug || Bug || Bug || Bug
| '''Calendar''' || TBD || {{bug|837671}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Contacts''' || TBD || Bug || Bug || Bug || Bug
| '''Camera''' || TBD || {{bug|1015385}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Dialer''' || TBD || Bug || Bug || Bug || Bug
| '''Clock''' || TBD || {{bug|837668}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Email''' || TBD || Bug || Bug || Bug || Bug
| '''Contacts''' || TBD || {{bug|1015388}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Gallery''' || TBD || Bug || Bug || Bug || Bug
| '''Dialer''' || TBD || {{bug|837668}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Marketplace''' || TBD || Bug || Bug || Bug || Bug
| '''Email''' || TBD || {{bug|837677}} || Bug || Bug || Bug || Bug || Bug
|-
| '''Settings''' || TBD || Bug || Bug || Bug || Bug
|-
| '''SMS''' || TBD || Bug || Bug || Bug || Bug
|-
|}
 
=== Milestone 3: Publication ===
'''Tracking Bug:''' TBD
 
{| class="wikitable"
! width="120px" |
! Target Release
! Published Results
! Documentation
|-
|-
| '''Browser''' || TBD || Bug || Bug
| '''FM Radio''' || TBD || {{bug|1015412}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Camera''' || TBD || Bug || Bug
| '''Gallery''' || TBD || {{bug|837674}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Contacts''' || TBD || Bug || Bug
| '''Marketplace''' || TBD || {{bug|837668}} || Bug || Bug || Bug || Bug || Bug
|-
|-
| '''Dialer''' || TBD || Bug || Bug
| '''Music''' || TBD || {{bug|837675}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Email''' || TBD || Bug || Bug
| '''Settings''' || TBD || {{bug|1015405}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Gallery''' || TBD || Bug || Bug
| '''SMS''' || TBD || {{bug|1015390}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
| '''Marketplace''' || TBD || Bug || Bug
| '''Template''' || TBD || {{bug|1015391}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''Settings''' || TBD || Bug || Bug
| '''Usage''' || TBD || {{bug|837678}} || N/A || Bug || Bug || Bug || Bug
|-
|-
| '''SMS''' || TBD || Bug || Bug
| '''Video''' || TBD || {{bug|837658}} || [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads Light] || Bug || Bug || Bug || Bug
|-
|-
|}
|}


<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Test_Template template]</small>
<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Metric_Template template]</small>

Latest revision as of 13:59, 1 February 2015

Launch Latency

Summary

Measures elapsed time from the first launch of an application to a series of defined startup events.

Usage

Test Cases

Launch each of the following apps from a cold state:

  • Browser
  • Calendar
  • Camera
  • Clock
  • Contacts
  • Dialer
  • Email
  • FM Radio
  • Gallery
  • Marketplace
  • Music
  • Settings
  • SMS
  • Template
  • Usage
  • Video

For each app, report the time elapsed to the following events:

  • moz-chrome-dom-loaded
  • moz-chrome-interactive
  • moz-app-visually-complete
  • moz-content-interative
  • moz-app-loaded

Precision

TBD

How to Run On-Demand

TBD

Published Results

FxOS 2.0

Flame
startup_>_moz-chrome-dom-loaded Datazilla
startup_>_moz-chrome-interactive Datazilla
startup_>_moz-app-visually-complete Datazilla
startup_>_moz-content-interative Datazilla
startup_>_moz-app-loaded Datazilla

FxOS 2.1

Flame
startup_>_moz-chrome-dom-loaded Datazilla
startup_>_moz-chrome-interactive Datazilla
startup_>_moz-app-visually-complete Datazilla
startup_>_moz-content-interative Datazilla
startup_>_moz-app-loaded Datazilla

References

Development

Design

Common Setup

  1. Set up workload
  2. Restart B2G process

All Cases

  1. Invoke launch programmatically
  2. Inject the Performance Helper and save start time
  3. App instrumentation throws timeline events at appropriate times
  4. Performance helper observes events
  5. Event emit times are compared with start time for time elapsed

Results

  • Result is time elapsed to each event
  • Execute test 30 times, for each event report mean, median, deviation

Project Roadmap

  • Lead: Eli Perelman
  • Tracking Bug: TBD
Milestone 1: Design
Test Design Validity Review
All Cases Bug Bug
Milestone 2: Test Milestone 3: Publication
Availability Instrumentation Workload On-Demand Test Results Review Published Results Documentation
Browser TBD bug 837668 Bug Bug Bug Bug Bug
Calendar TBD bug 837671 Bug Bug Bug Bug Bug
Camera TBD bug 1015385 N/A Bug Bug Bug Bug
Clock TBD bug 837668 N/A Bug Bug Bug Bug
Contacts TBD bug 1015388 Light Bug Bug Bug Bug
Dialer TBD bug 837668 Light Bug Bug Bug Bug
Email TBD bug 837677 Bug Bug Bug Bug Bug
FM Radio TBD bug 1015412 N/A Bug Bug Bug Bug
Gallery TBD bug 837674 Light Bug Bug Bug Bug
Marketplace TBD bug 837668 Bug Bug Bug Bug Bug
Music TBD bug 837675 Light Bug Bug Bug Bug
Settings TBD bug 1015405 N/A Bug Bug Bug Bug
SMS TBD bug 1015390 Light Bug Bug Bug Bug
Template TBD bug 1015391 N/A Bug Bug Bug Bug
Usage TBD bug 837678 N/A Bug Bug Bug Bug
Video TBD bug 837658 Light Bug Bug Bug Bug

template

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_OS/Performance/Automation/Launch_Latency&oldid=1052495"