Firefox OS/Performance/Automation/Subsequent Launch Latency: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (Lakrits moved page FirefoxOS/Performance/Automation/Subsequent Launch Latency to Firefox OS/Performance/Automation/Subsequent Launch Latency: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to...)
 
(9 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Subsequent Launch Latency =
= Subsequent Launch Latency =
== Summary ==
== Summary ==


Measures elapsed time from a launch of an application already in memory to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].
Measures elapsed time from a launch of an application already in memory to a series of defined [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines startup events].


== Precision ==
== Usage ==


TBD
=== Test Cases ===
 
== Scenarios ==


Launch each of the following apps from a warm state:
Launch each of the following apps from a warm state:


* Browser
* Browser
* Calendar
* Camera
* Camera
* Clock
* Contacts
* Contacts
* Dialer
* Dialer
* Email
* Email
* FM Radio
* Gallery
* Gallery
* Marketplace
* Marketplace
* Music
* Settings
* Settings
* SMS
* SMS
* Template
* Usage
* Video


For each app, report the time elapsed to the following events:
For each app, report the time elapsed to the following events:
Line 30: Line 36:
* moz-app-loaded
* moz-app-loaded


== Dashboards ==
=== Precision ===
=== 2.0 ===
 
TBD
 
=== How to Run On-Demand ===
 
TBD
 
=== Published Results ===
 
==== FxOS 2.0 ====


{| class="wikitable"
{| class="wikitable"
! Test
!  
! Flame
! Flame
|-
|-
Line 49: Line 64:
|}
|}


=== 2.1 ===
==== FxOS 2.1 ====


{| class="wikitable"
{| class="wikitable"
! Test
!  
! Flame
! Flame
|-
|-
Line 67: Line 82:
|}
|}


== Procedure ==
=== References ===
 
* [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines Firefox OS App Responsiveness Guidelines]


''This needs more detail.''
== Development ==


# Clear app from memory
=== Design ===
# Inject launch command
 
These tests use the same workload and instrumentation as the [https://wiki.mozilla.org/FirefoxOS/Performance/Automation/Launch_Latency Launch Latency] tests.
 
==== Common Setup ====
# Set up workload
# Restart B2G Process
# Invoke launch programmatically
# Close app
# Close app
# Inject launch command again
# Monitor app events for each of the five events under Scenarios
# Time elapsed is the time between that event and ''start time?''


== References ==
==== All Cases ====
# Invoke launch programmatically
# Inject the Performance Helper and save start time
# App instrumentation throws timeline events at appropriate times
# Performance helper observes events
# Event emit times are compared with start time for time elapsed


TBD
==== Results ====
* Result is time elapsed to each event
* Execute test 30 times, for each event report mean, median, deviation


== Roadmap ==
=== Project Roadmap ===


=== Overview ===
==== Overview ====


* '''Lead:''' TBD
* '''Lead:''' Eli Perelman
* '''Tracking Bug:''' TBD
* '''Tracking Bug:''' TBD
=== Milestone 1: Design ===
'''Tracking Bug:''' TBD


{| class="wikitable"
{| class="wikitable"
! width="120px" |  
! width="120px" |
! Target Release
! colspan="2" | Milestone 1: Design
! Test Design  
|-
!
! Test Design
! Validity Review
! Validity Review
|-
|-
| '''All''' || TBD || Bug || Bug
| '''All Cases''' || Bug || Bug
|-
|-
|}
|}
=== Milestone 2: Test ===
'''Tracking Bug:''' TBD


{| class="wikitable"
{| class="wikitable"
! width="120px" |
! width="120px" |
! Target Release
!
! Instrumentation
! colspan="2" | Milestone 2: Test
! Workloads
! colspan="2" | Milestone 3: Publication
! Single-run test
|-
!
! Availability
! On-Demand Test
! Results Review
! Results Review
! Published Results
! Documentation
|-
|-
| '''Browser''' || TBD || Bug || Bug || Bug || Bug
| '''Browser''' || TBD || Bug || Bug || Bug || Bug
|-
| '''Calendar''' || TBD || Bug || Bug || Bug || Bug
|-
|-
| '''Camera''' || TBD || Bug || Bug || Bug || Bug
| '''Camera''' || TBD || Bug || Bug || Bug || Bug
|-
| '''Clock''' || TBD || Bug || Bug || Bug || Bug
|-
|-
| '''Contacts''' || TBD || Bug || Bug || Bug || Bug
| '''Contacts''' || TBD || Bug || Bug || Bug || Bug
Line 122: Line 154:
|-
|-
| '''Email''' || TBD || Bug || Bug || Bug || Bug
| '''Email''' || TBD || Bug || Bug || Bug || Bug
|-
| '''FM Radio''' || TBD || Bug || Bug || Bug || Bug
|-
|-
| '''Gallery''' || TBD || Bug || Bug || Bug || Bug
| '''Gallery''' || TBD || Bug || Bug || Bug || Bug
|-
|-
| '''Marketplace''' || TBD || Bug || Bug || Bug || Bug
| '''Marketplace''' || TBD || Bug || Bug || Bug || Bug
|-
| '''Music''' || TBD || Bug || Bug || Bug || Bug
|-
|-
| '''Settings''' || TBD || Bug || Bug || Bug || Bug
| '''Settings''' || TBD || Bug || Bug || Bug || Bug
Line 131: Line 167:
| '''SMS''' || TBD || Bug || Bug || Bug || Bug
| '''SMS''' || TBD || Bug || Bug || Bug || Bug
|-
|-
|}
| '''Template''' || TBD || Bug || Bug || Bug || Bug
 
=== Milestone 3: Publication ===
'''Tracking Bug:''' TBD
 
{| class="wikitable"
! width="120px" |
! Target Release
! Published Results
! Documentation
|-
| '''Browser''' || TBD || Bug || Bug
|-
| '''Camera''' || TBD || Bug || Bug
|-
| '''Contacts''' || TBD || Bug || Bug
|-
| '''Dialer''' || TBD || Bug || Bug
|-
| '''Email''' || TBD || Bug || Bug
|-
| '''Gallery''' || TBD || Bug || Bug
|-
| '''Marketplace''' || TBD || Bug || Bug
|-
|-
| '''Settings''' || TBD || Bug || Bug
| '''Usage''' || TBD || Bug || Bug || Bug || Bug
|-
|-
| '''SMS''' || TBD || Bug || Bug
| '''Video''' || TBD || Bug || Bug || Bug || Bug
|-
|-
|}
|}


<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Test_Template template]</small>
<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Metric_Template template]</small>

Latest revision as of 13:59, 1 February 2015

Subsequent Launch Latency

Summary

Measures elapsed time from a launch of an application already in memory to a series of defined startup events.

Usage

Test Cases

Launch each of the following apps from a warm state:

  • Browser
  • Calendar
  • Camera
  • Clock
  • Contacts
  • Dialer
  • Email
  • FM Radio
  • Gallery
  • Marketplace
  • Music
  • Settings
  • SMS
  • Template
  • Usage
  • Video

For each app, report the time elapsed to the following events:

  • moz-chrome-dom-loaded
  • moz-chrome-interactive
  • moz-app-visually-complete
  • moz-content-interative
  • moz-app-loaded

Precision

TBD

How to Run On-Demand

TBD

Published Results

FxOS 2.0

Flame
startup_>_moz-chrome-dom-loaded
startup_>_moz-chrome-interactive
startup_>_moz-app-visually-complete
startup_>_moz-content-interative
startup_>_moz-app-loaded

FxOS 2.1

Flame
startup_>_moz-chrome-dom-loaded
startup_>_moz-chrome-interactive
startup_>_moz-app-visually-complete
startup_>_moz-content-interative
startup_>_moz-app-loaded

References

Development

Design

These tests use the same workload and instrumentation as the Launch Latency tests.

Common Setup

  1. Set up workload
  2. Restart B2G Process
  3. Invoke launch programmatically
  4. Close app

All Cases

  1. Invoke launch programmatically
  2. Inject the Performance Helper and save start time
  3. App instrumentation throws timeline events at appropriate times
  4. Performance helper observes events
  5. Event emit times are compared with start time for time elapsed

Results

  • Result is time elapsed to each event
  • Execute test 30 times, for each event report mean, median, deviation

Project Roadmap

Overview

  • Lead: Eli Perelman
  • Tracking Bug: TBD
Milestone 1: Design
Test Design Validity Review
All Cases Bug Bug
Milestone 2: Test Milestone 3: Publication
Availability On-Demand Test Results Review Published Results Documentation
Browser TBD Bug Bug Bug Bug
Calendar TBD Bug Bug Bug Bug
Camera TBD Bug Bug Bug Bug
Clock TBD Bug Bug Bug Bug
Contacts TBD Bug Bug Bug Bug
Dialer TBD Bug Bug Bug Bug
Email TBD Bug Bug Bug Bug
FM Radio TBD Bug Bug Bug Bug
Gallery TBD Bug Bug Bug Bug
Marketplace TBD Bug Bug Bug Bug
Music TBD Bug Bug Bug Bug
Settings TBD Bug Bug Bug Bug
SMS TBD Bug Bug Bug Bug
Template TBD Bug Bug Bug Bug
Usage TBD Bug Bug Bug Bug
Video TBD Bug Bug Bug Bug

template

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_OS/Performance/Automation/Subsequent_Launch_Latency&oldid=1052503"