canmove, Confirmed users
2,041
edits
B2G/QA/Automation/UI/Strategy/Integration vs End to end: Difference between revisions
B2G/QA/Automation/UI/Strategy/Integration vs End to end (view source)
Revision as of 02:27, 12 February 2015
, 12 February 2015→The Solution
| Line 30: | Line 30: | ||
== Two different suites of UI automation == | == Two different suites of UI automation == | ||
* | * Gaia Integration | ||
** Must be quick-running with no non-deterministic factors. | ** Must be quick-running with no non-deterministic factors. | ||
** Results must be absolutely unambiguous. | ** Results must be absolutely unambiguous. | ||
| Line 40: | Line 40: | ||
** Runs on B2G Desktop currently, will run on device as well soon. | ** Runs on B2G Desktop currently, will run on device as well soon. | ||
* | * Gaia UAT | ||
** Can have longer-running tests and have reasonable amounts of fragility due to non-determinism | ** Can have longer-running tests and have reasonable amounts of fragility due to non-determinism | ||
** Results can be occasionally ambiguous as a trade off for higher coverage | ** Results can be occasionally ambiguous as a trade off for higher coverage | ||
| Line 54: | Line 54: | ||
The main differences are: | The main differences are: | ||
* | * Gaia Integration tests run faster and are generally more isolated. | ||
* UATs have more coverage | * Gaia UATs have more coverage due to less rules. | ||
* | * Gaia Integration tests can never fail unless there's a defective code change. They must be unambiguous. | ||
* UATs can still fail non-deterministically if they are reliable enough to be a net gain. | * Gaia UATs can still fail non-deterministically if they are reliable enough to be a net gain. | ||
* | * Gaia Integration tests are maintained by development to give themselves confidence in code changes. | ||
* UATs are maintained by QA to replace or extend manual regression testing. | * Gaia UATs are maintained by QA to replace or extend manual regression testing. | ||
* | * Gaia Integration tests frequently test fragments of UI behavior and may be based on mocks or other low-level objects. | ||
* UATs are always written as complete user-like scenarios, and generally operate and verify as the user would. | * Gaia UATs are always written as complete user-like scenarios, and generally operate and verify as the user would. | ||
* It is more important that | * It is more important that Gaia Integration tests be solid than that they're complete. Any incremental gain is valuable. | ||
* It is more important that UATs be complete than solid. All acceptance criteria must be tested to accept a build. | * It is more important that Gaia UATs be complete than solid. All acceptance criteria must be tested to accept a build. | ||
== Ownership and Overlap == | == Ownership and Overlap == | ||
| Line 73: | Line 73: | ||
Ownership is separated because these differences can lead to variant needs in terms of breadth and depth of verification. | Ownership is separated because these differences can lead to variant needs in terms of breadth and depth of verification. | ||
Tests can and should overlap between | Tests can and should overlap between Gaia Integration and Gaia UAT. It is unacceptable process complication for QA to expect developers to always consult their needs and vice versa for every change. If the suites are united, it's all too easy for one group to inadvertently make a change that damages the purpose of the other. For acceptance, in particular, this is too risky. | ||
Avoiding this requires separate tests, even to the point where there might be tests whose code is entirely duplicated between the two suites. | Avoiding this requires separate tests, even to the point where there might be tests whose code is entirely duplicated between the two suites. | ||