Firefox OS/Performance/Automation/Launch Latency: Difference between revisions
| Line 8: | Line 8: | ||
''This would be filled in with the estimated precision of the test based on result variability and what we know about the precision of the measurement methodology. This is different than accuracy, and represents how much "scatter" of the results around the true result we might expect. | ''This would be filled in with the estimated precision of the test based on result variability and what we know about the precision of the measurement methodology. This is different than accuracy, and represents how much "scatter" of the results around the true result we might expect. | ||
''We might estimate a precision of 10ms or we might estimate a precision of 100ms. The latter would imply very different | ''We might estimate a precision of 10ms or we might estimate a precision of 100ms. The latter would imply very different criteria before we could assume a regression.'' | ||
== Scenarios == | == Scenarios == | ||
Revision as of 03:41, 21 July 2014
Launch Latency
Summary
Measures elapsed time from the first launch of an application to a series of defined startup events.
Precision
This would be filled in with the estimated precision of the test based on result variability and what we know about the precision of the measurement methodology. This is different than accuracy, and represents how much "scatter" of the results around the true result we might expect.
We might estimate a precision of 10ms or we might estimate a precision of 100ms. The latter would imply very different criteria before we could assume a regression.
Scenarios
Launch each of the following apps from a cold state:
- Browser
- Camera
- Contacts
- Dialer
- Gallery
- Marketplace
- Settings
- SMS
For each app, report the time elapsed to the following events:
- moz-chrome-dom-loaded
- moz-chrome-interactive
- moz-app-visually-complete
- moz-content-interative
- moz-app-loaded
Dashboards
This should be a series of links to dashboards per branch, device, and test. Unfortunately, Datazilla doesn't seem to allow direct linking without embedding revisions, specific implemented apps, and other time-specific artifacts in the URL. This is a real problem.
2.0
| Test | Flame |
|---|---|
| startup_>_moz-chrome-dom-loaded | link |
| startup_>_moz-chrome-interactive | link |
| startup_>_moz-app-visually-complete | link |
| startup_>_moz-content-interative | link |
| startup_>_moz-app-loaded | link |
2.1
| Test | Flame |
|---|---|
| startup_>_moz-chrome-dom-loaded | link |
| startup_>_moz-chrome-interactive | link |
| startup_>_moz-app-visually-complete | link |
| startup_>_moz-content-interative | link |
| startup_>_moz-app-loaded | link |
Procedure
This needs more detail.
- Clear app from memory
- Inject launch command
- Monitor app events for each of the five events under Scenarios
- Time elapsed is the time between that event and start time?
References
TBD
Roadmap
Overview
- Lead: Eli Perelman
- Tracking Bug: TBD
Milestone 1: Design
Tracking Bug: TBD
| Target Release | Test Design | Validity Review | |
|---|---|---|---|
| All | TBD | Bug | Bug |
Milestone 2: Test
Tracking Bug: TBD
| Target Release | Instrumentation | Workloads | Single-run test | Results Review | |
|---|---|---|---|---|---|
| Browser | TBD | Bug | Bug | Bug | Bug |
| Camera | TBD | Bug | Bug | Bug | Bug |
| Contacts | TBD | Bug | Bug | Bug | Bug |
| Dialer | TBD | Bug | Bug | Bug | Bug |
| TBD | Bug | Bug | Bug | Bug | |
| Gallery | TBD | Bug | Bug | Bug | Bug |
| Marketplace | TBD | Bug | Bug | Bug | Bug |
| Settings | TBD | Bug | Bug | Bug | Bug |
| SMS | TBD | Bug | Bug | Bug | Bug |
Milestone 3: Publication
Tracking Bug: TBD
| Target Release | Published Results | Documentation | |
|---|---|---|---|
| Browser | TBD | Bug | Bug |
| Camera | TBD | Bug | Bug |
| Contacts | TBD | Bug | Bug |
| Dialer | TBD | Bug | Bug |
| TBD | Bug | Bug | |
| Gallery | TBD | Bug | Bug |
| Marketplace | TBD | Bug | Bug |
| Settings | TBD | Bug | Bug |
| SMS | TBD | Bug | Bug |