canmove, Confirmed users
2,041
edits
B2G/QA/Automation/UI/Best Practices: Difference between revisions
→Tests that check functional behavior
| (17 intermediate revisions by the same user not shown) | |||
| Line 8: | Line 8: | ||
= Synchronization = | = Synchronization = | ||
* Waits should almost always be conditional, using [http://marionette-client.readthedocs.org/en/latest/reference.html#wait | * Waits should almost always be conditional, using Marionette [http://marionette-client.readthedocs.org/en/latest/reference.html#wait Wait().until()] constructs. | ||
* Conversely, any sort of hard sleep() type construct is discouraged. These cause tests to have a longer best-case execution time. | * Conversely, any sort of hard sleep() type construct is discouraged. These cause tests to have a longer best-case execution time, instead of adapting their runtime to the best possible allowed by the AUT. | ||
= Abstraction = | = Abstraction = | ||
| Line 25: | Line 25: | ||
* Will consist of a string of UI calls such as calendar.newEntryButton.tap() and calendar.summary.text == ... | * Will consist of a string of UI calls such as calendar.newEntryButton.tap() and calendar.summary.text == ... | ||
* Should not delegate strings of UI calls used in a test to functional-style Page Objects or similar high level abstraction mechanisms. They own that sequence. | * Should not delegate strings of UI calls used in a test to functional-style Page Objects or similar high level abstraction mechanisms. They own that sequence. | ||
* Will be very fragile, as any change in UI assumption will require changes to all tests making that assumption. | |||
* Will be very fragile, as any change in UI assumption will | |||
* Should therefore be kept very short and isolated, with no extended sequences. Ideally, one changed assumption = one broken test. | * Should therefore be kept very short and isolated, with no extended sequences. Ideally, one changed assumption = one broken test. | ||
* Will not generally correspond with a traditionally-scripted QA test. They'll be much shorter. | * Should still abstract selectors as these change frequently and aren't directly important to the test. | ||
* Will not generally correspond with a traditionally-scripted QA test. They'll be much shorter and simpler, more like a single test step. | |||
* Should have almost no logic whatsoever. | * Should have almost no logic whatsoever. | ||
| Line 49: | Line 49: | ||
* Should not be concerned with the UI or API details of how they perform their actions or verify results. | * Should not be concerned with the UI or API details of how they perform their actions or verify results. | ||
* Should not call or verify UI controls or back-end APIs directly in test code. | * Should not call or verify UI controls or back-end APIs directly in test code. | ||
* Should | * Should encapsulate UI actions and checks in [http://martinfowler.com/bliki/PageObject.html Page (aka App or Region) Objects]. The Page Object should be the only place UI is used directly. | ||
* Encapsulate API details in objects or reusable functions as well. | |||
* Should favor verifications through UI over verifications of | * Should favor verifications round trip through UI over verifications of back end APIs. They should look for what the user would look for. | ||
* Will consist of a string of functions at high-level, such as calendar.createEntry(...) and calendar.getEntry() == defaultEntry. | * Will consist of a string of functions at high-level, such as calendar.createEntry(...) and calendar.getEntry() == defaultEntry. | ||
* Will be longer and more complex, so need to be at a higher level of abstraction to promote reuse and keep clarity. | * Will be longer and more complex, so need to be at a higher level of abstraction to promote reuse and keep clarity. | ||
| Line 60: | Line 60: | ||
* Tests are frequently the only documentation of application behavior | * Tests are frequently the only documentation of application behavior | ||
* Any extraneous detail in the test interferes with clarity and ability to review for correctness. | |||
* Test code should be absolutely explicit about behavior at their functional level of [[#Abstraction|abstraction]]. | * Test code should be absolutely explicit about behavior at their functional level of [[#Abstraction|abstraction]]. | ||
* Test code should be | * Test code should be implicit about behavior below their functional level of [[#Abstraction|abstraction]]. | ||
* Reading a test should tell you as closely as possible what the test ''means'' to do, no more and no less. | * Reading a test should tell you as closely as possible what the test ''means'' to do, no more and no less. | ||
= Logic = | = Logic = | ||
* Generally, minimize logic in tests. Tests do one thing and should require branching. | * Generally, minimize logic in tests. Tests do one thing and should rarely require branching on decisions. | ||
* However, there are exceptions. A much larger discussion on this can be found [https://github.com/geoelectric/ui_auto_best_practice/blob/master/uiabp.md here]. | * However, there are exceptions. A much larger discussion on this can be found [https://github.com/geoelectric/ui_auto_best_practice/blob/master/uiabp.md here]. | ||
= State = | |||
* The test harness should always start a test from a single known state. | |||
* Tests should not depend on other tests for their state. Every test should stand alone, and should be runnable in any order. | |||
* Tests should not try to keep application state, except to use as a comparison for verification later. | |||
Resetting the AUT to a known state is the single most important functionality in a test suite or harness. It must be absolutely correct, and it must run as fast as possible. If it is unreliable, the tests are unreliable. If it is slow, testers will start combining tests together to avoid overhead between tests and tests will become unmaintainable. | |||
When given a choice between putting efforts into tests and putting efforts into improving the reset/known-state handling of the harness, pick the harness every time. | |||
= Programming Best Practices = | = Programming Best Practices = | ||
| Line 75: | Line 85: | ||
* Optimize for correctness to expected application behavior and clarity, in that order. The job of the test is both to verify correctly and communicate correct behavior to the reader. | * Optimize for correctness to expected application behavior and clarity, in that order. The job of the test is both to verify correctly and communicate correct behavior to the reader. | ||
* Don't share ownership of test details. If a test needs a particular sequence in a particular order, the test should own that sequence in its own code, even if some other test also owns a similar or same sequence elsewhere. Don't risk letting modifications to one test make another test incorrect. | * Don't share ownership of test details. If a test needs a particular sequence in a particular order, the test should own that sequence in its own code, even if some other test also owns a similar or same sequence elsewhere. Don't risk letting modifications to one test make another test incorrect. | ||
** UI mechanics tests that must perform a specific sequence of UI operations in a specific order are performing ''macros'', a sequence of UI actions. These are not candidates for moving into functions, as functions are not allowed to expose their implementation or they break integrity. Macros can be reused, but should be kept local to the test file. | ** UI mechanics tests that must perform a specific sequence of UI operations in a specific order are performing ''macros'', a sequence of UI actions. These are not candidates for moving into shared functions, as functions are not allowed to expose details of their implementation or they break integrity. Macros can be reused, but should be kept local to the test file. | ||
** Functional tests that abstract over UI still own a sequence of functional calls. Similarly, any reuse of that sequence should be kept local to the test file. | ** Functional tests that abstract over UI still own a sequence of functional calls. Similarly, any reuse of that sequence should be kept local to the test file. | ||
* Do share ownership of anything that is not a test detail, including (and especially) almost any particular task done in setup and cleanup. Know your level of [[#Abstraction|abstraction]]. | * Do share ownership of anything that is not a test detail, including (and especially) almost any particular task done in setup and cleanup. Know your level of [[#Abstraction|abstraction]]. | ||
* Sometimes setup/cleanup needs a function and the test needs a macro that do the same thing. Sharing code for these is risky. It's usually better to let setup/cleanup use | * Sometimes setup/cleanup needs a function and the test needs a macro that do the same thing. Sharing code for these is risky. It's usually better to let setup/cleanup use a highly robust function, preferably without involving UI, that you can change at will to make faster or more robust later without risking test coverage. | ||
* Balance SPOT with your organizational structure. If it's not natural for two groups to communicate, those groups should not have shared ownership of code. | * Balance SPOT with your organizational structure. If it's not natural for two groups to communicate, those groups should not have shared ownership of code. Either refactor shared code into a module with single ownership and consumed by the others as a version-pinned library, or both groups should maintain their own copies. SPOT only helps when you all talk. Keep in mind more than a few people needing to talk to make a given change is an [http://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_mythical_man-month antipattern]. | ||
= Test Runs = | = Test Runs = | ||
* Separate stable and unstable tests. Stable test failures can be assumed to be correct, and can be escalated automatically. | * Separate stable and unstable tests. Stable test failures can be assumed to be correct, and can be escalated automatically. | ||
* Keep test runs green by [../ | * Keep test runs green by [[../Xfail and Disable|pruning known failures]]. It's important to know which tests have just started failing today. | ||
* All test failures must be retested manually, even known intermittents. | * All test failures must be retested manually, even known intermittents. | ||
* Tests that are disabled must be re-added to manual test runs, or the coverage will be dropped. | * Tests that are disabled must be re-added to manual test runs, or the coverage will be dropped. | ||