B2G/QA/Automation/UI/Strategy/Integration vs End to end: Difference between revisions

In particular, integration automation run before landing a commit is generally subject to a series of restrictions meant to ensure that the test results are returned quickly and in an unambiguous fashion. Those restrictions reduce the usefulness of the automation for use in acceptance testing, which must often accept a higher level of fragility and longer runtime to be more comprehensive.

Commit integration automation is incomplete and shallow, trading expedience for coverage, and is generally written from a developer's perspective to ensure the behavior of what they did write--which might not actually meet the requirements of what they should have written. Since the tests tend to be written against a particular section of the SUT, broad interactions may go uncovered. Large portions of the SUT may go untested because they necessarily rely on unreliable externals. As such, it frequently falls short of the needs of acceptance.  

Conversely, acceptance automation is too slow, device-bound, and sometimes has spurious failures due to non-determinism inherent in the SUT and its externals. As the user scenarios it tests often bridge multiple parts of the system, it's very fragile with poor isolation. The need to rely on user-like behavior and check after every step slows it down. The necessary scope means it often can't be written adequately until entire subsystems are in place. So it's often wholly unsuitable for the incremental growth, quick land-or-not decisions, and debugging assistance that a good continuous integration suite provides.