canmove, Confirmed users
2,041
edits
Firefox OS/Performance/Automation: Difference between revisions
→Overview
| Line 38: | Line 38: | ||
* There are many moving parts to pull together for a single test, and often different owners for the moving parts. | * There are many moving parts to pull together for a single test, and often different owners for the moving parts. | ||
* The tests are sometimes more invasive to the system-under-test than a functional test would be. This requires extreme caution; tests can break functionality or even skew their own results. | * The tests are sometimes more invasive to the system-under-test than a functional test would be. This requires extreme caution; tests can break functionality or even skew their own results. | ||
* It's very easy to measure the wrong thing, get reasonable looking results, and trust a faulty test. | * Performance tests can be maintenance-intensive. Permanent instrumentation can make the maintenance of the tested code harder too. It's important the test be worth the risk and cost. | ||
* It's very easy to measure the right thing but not control the environment correctly, and trust a faulty test. | * It's very easy to measure the wrong thing, get reasonable looking results, and trust a faulty test or receive false alerts. | ||
* It's very easy to measure the right thing but not control the environment correctly, and trust a faulty test or receive false alerts. | |||
* Performance bugs are hugely expensive to triage, diagnose, and fix; false alerts have significant costs. | * Performance bugs are hugely expensive to triage, diagnose, and fix; false alerts have significant costs. | ||
* If performance tests | * Once the test is running, people don't tend to look for the problems themselves. Tests that don't alert when they should let performance issues slip into field and cause perception of our quality to be lower. | ||
* If performance tests prove untrustworthy, the metric in general will sometimes be written off and even after repair the results may be treated with low priority. | |||
In short, performance tests are a mission-critical system, and require a mission-critical approach. They must be correct, and their behavioral and statistical characteristics need to be well-known. | In short, performance tests are a mission-critical system, and require a mission-critical approach. They must be correct, and their behavioral and statistical characteristics need to be well-known. | ||
Our workflow is described in a series of deliverables. | |||
With only a couple of exceptions, each deliverable depends on the last to be completed before it may be started. Reviews for validity and result repeatability are inline with other deliverables to ensure each test brings positive value to Firefox OS. | |||
=== Deliverables === | === Deliverables === | ||