canmove, Confirmed users
2,041
edits
Firefox OS/Performance/Automation: Difference between revisions
→General Development Workflow
| Line 29: | Line 29: | ||
A performance automation metric consists of a type of measurement applied to a series of test cases. | A performance automation metric consists of a type of measurement applied to a series of test cases. | ||
Depending on the metric, these may be one basic test scenario applied to a number of apps (such as launch latency testing) or a number of | Depending on the metric, these may be one basic test scenario applied to a number of apps (such as launch latency testing) or a number of distinctly designed test cases (such as scenarios for memory or power consumption testing). | ||
The workflow for developing a performance automation test case must be more structured than a standard unit-testing or functional-testing development effort. Reasons are: | |||
* There are many moving parts to pull together for a single test, and often different owners for the moving parts. | |||
* The tests are sometimes more invasive to the system-under-test than a functional test would be. This requires extreme caution; tests can break functionality or even skew their own results. | |||
* It's very easy to measure the wrong thing, get reasonable looking results, and trust a faulty test. | |||
* It's very easy to measure the right thing but not control the environment correctly, and trust a faulty test. | |||
* Performance bugs are hugely expensive to triage, diagnose, and fix; false alerts have significant costs. | |||
* If performance tests do prove untrustworthy, the metric in general will sometimes be written off and even after repair the results will be treated with low priority. | |||
In short, performance tests are a mission-critical system, and require a mission-critical approach. They must be correct, and their behavioral and statistical characteristics need to be well-known. | |||
Deliverables | |||
; Design : Detailed description of the test case's setup needs and procedure, the test procedure, execution details, and results to deliver. This might be for a common scenario, or might be per-case. | ; Design : Detailed description of the test case's setup needs and procedure, the test procedure, execution details, and results to deliver. This might be for a common scenario, or might be per-case. | ||