3,860
edits
Firefox OS/Performance/Automation: Difference between revisions
Firefox OS/Performance/Automation (view source)
Revision as of 13:59, 1 February 2015
, 1 February 2015Lakrits moved page FirefoxOS/Performance/Automation to Firefox OS/Performance/Automation: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the spelling of its name is consiste...
m (Lakrits moved page FirefoxOS/Performance/Automation to Firefox OS/Performance/Automation: The official spelling of "Firefox OS" leaves a space between the two parts of the name. It's easier to find a page if the spelling of its name is consiste...) |
|||
| (9 intermediate revisions by 2 users not shown) | |||
| Line 5: | Line 5: | ||
* Provide developers with timely information regarding the performance of their latest code changes | * Provide developers with timely information regarding the performance of their latest code changes | ||
* Provide product and quality owners with enough information to know whether products will be acceptable | * Provide product and quality owners with enough information to know whether products will be acceptable | ||
* Give piece of mind when | * Give piece of mind when Firefox OS performs well | ||
* Inspire appropriate urgency when | * Inspire appropriate urgency when Firefox OS does not | ||
== Tools == | |||
* [[/Tools|Automation Tools]] | |||
== Tests == | == Tests == | ||
| Line 12: | Line 15: | ||
=== Instrumented === | === Instrumented === | ||
* [[/Launch Latency|Launch Latency]] | * [[/Launch Latency|Launch Latency]] | ||
* [[/Subsequent Launch Latency| | * [[/Subsequent Launch Latency|Relaunch Latency]] | ||
* [[/Frame Rate|Frame Rate]] | * [[/Frame Rate|Frame Rate]] | ||
* [[/Frame Uniformity|Frame Uniformity]] | * [[/Frame Uniformity|Frame Uniformity]] | ||
| Line 25: | Line 28: | ||
<small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Metric_Template template]</small> | <small>[http://wiki.mozilla.org/FirefoxOS/Performance/Automation/Metric_Template template]</small> | ||
== | == Development Strategy == | ||
=== Overview === | === Overview === | ||
| Line 46: | Line 49: | ||
In short, performance tests are a mission-critical system, and require a mission-critical approach. They must be correct, and their behavioral and statistical characteristics need to be well-known. The consequences of having bad performance tests can be worse than the consequences of having no test at all. | In short, performance tests are a mission-critical system, and require a mission-critical approach. They must be correct, and their behavioral and statistical characteristics need to be well-known. The consequences of having bad performance tests can be worse than the consequences of having no test at all. | ||
Our workflow is described in a series of deliverables | Our workflow is described in a series of intermediate deliverables. Reviews for validity and result repeatability are inline with other deliverables to ensure each test brings positive value to Firefox OS. | ||
With only a couple of exceptions, each deliverable depends on the last to be completed before it may be started. | With only a couple of exceptions, each deliverable depends on the last to be completed before it may be started. | ||
| Line 55: | Line 58: | ||
; Validity Review : Review by test stakeholders for whether the test case design meets requirements and will measure the right thing. Tests that measure the wrong thing can still give repeatable results, so it is important to review the initial design. | ; Validity Review : Review by test stakeholders for whether the test case design meets requirements and will measure the right thing. Tests that measure the wrong thing can still give repeatable results, so it is important to review the initial design. | ||
; Workload : Data (a fixture) to be preloaded into the system under test prior to execution. There are default workflows described at [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads MDN], but some tests may need special-case workflows to be developed. | ; Workload : Data (a fixture) to be preloaded into the system under test prior to execution. There are default workflows described at [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Reference_Workloads MDN], but some tests may need special-case workflows to be developed. | ||
; Instrumentation : Code either injected | ; Instrumentation : Code either injected into or permanently added to Firefox OS or its applications in order to let automation monitor performance details. Some tests do not need this. | ||
; On-Demand Test : Test that can be run by a developer on command line, try server, or some other way to test a set of arbitrary code. | ; On-Demand Test : Test that can be run by a developer on command line, try server, or some other way to test a set of arbitrary code. | ||
; Results Review : Review by test stakeholders of a set of results of the implemented test. | ; Results Review : Review by test stakeholders of a set of results of the implemented test. We sanity check for whether the results look valid (a result frame rate of 70fps on a 60Hz screen would fail this) and whether they are repeatable (reasonably grouped, assuming no code changes). This is also where we can determine or finalize an estimate of test precision, based on real results. | ||
; Published Results : Automated results published to a dashboard or similar display. This usually also implies having | ; Published Results : Automated results published to a dashboard or similar display. This usually also implies having an automated triggering and execution mechanism (nightly, on-commit, etc.) to generate them. | ||
; Documentation : Finalized documentation on wiki and/or MDN of the test case and its usage. | ; Documentation : Finalized documentation on this wiki and/or MDN of the test case and its usage. | ||
=== Illustration === | === Illustration === | ||