B2G/QA/Automation/UI/Integration suite criteria: Difference between revisions

From MozillaWiki
< B2G‎ | QA‎ | Automation‎ | UI
Jump to navigation Jump to search
No edit summary
(Renaming acceptance to end-to-end)
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''''This document is currently in work in progress'''''
In order to make a clearer distinction between the [https://wiki.mozilla.org/B2G/QA/Automation/UI/Strategy/Integration_vs_End_to_end#The_Solution Integration suite and the End-to-end one], here is a list which helps to define where a test belongs depending on its nature.
 
In order to make a clearer distinction between the [https://wiki.mozilla.org/B2G/QA/Automation/UI/Strategy/Integration_vs_Acceptance#The_Solution Integration suite and the Acceptance one], here is a list which helps to define where a test belongs depending on its nature.


= General considerations =
= General considerations =


Should be in Integration suite rather than in the Acceptance one:
Should be in Integration suite rather than in the End-to-end one:
* Tests which require to mock an actor of the system
* Tests that don't represent a whole user story, such as boundary/edge cases, negative cases, and highly specific or unusual bug verifications.
* General edge cases tests *should we use the term negative test case instead?*
* Tests that perform small, isolated checks of the UI (such as a test that only verifies the sort order of a table)
* Tests coming with a regression fix with unusual STR
* Tests that use mocks to simulate a portion of the system
* Tests that verify something which is invisible for an end-user (example: "Is the gaia setting "ril.data.enabled" correctly set?")
* Tests that validate binding between the UI state and back end state
* Tests that perform an action which can't be done by an end-user (example: )
* Tests that verify something which is invisible for an end-user (example: "Is the gaia setting 'ril.data.enabled' correctly set?"). An exception could be made for an End-to-end test if this verification is done during the setUp/tearDown phases.


Should not be in Integration suite:
Should not be in Integration suite:
Line 16: Line 14:
* Tests that check a device specific issue.
* Tests that check a device specific issue.
* Tests that check the behavior with an external service without being mocked (like a test which interacts with the actual Gmail).
* Tests that check the behavior with an external service without being mocked (like a test which interacts with the actual Gmail).
* Tests that takes too long to be executed (more than 120 *other value?* seconds)
* Tests that takes too long to be executed.


= Firefox OS current specific considerations =  
= Firefox OS current specific considerations =  
Line 23: Line 21:
* relies on hardware which can't be currently mocked (like Camera, SIM/DSDS, FM antenna)
* relies on hardware which can't be currently mocked (like Camera, SIM/DSDS, FM antenna)
* depends on external website/service which can't be mocked (as Carrier network, Firefox Account, some Email providers)
* depends on external website/service which can't be mocked (as Carrier network, Firefox Account, some Email providers)
* Is blocked by a bug in b2g desktop, for instance:
* Is blocked by a bug in b2g desktop, (for instance: [https://bugzilla.mozilla.org/show_bug.cgi?id=1008930 MP4/3GP not recognized], [https://bugzilla.mozilla.org/show_bug.cgi?id=1116749 emergency dialer cannot be open])
** Airplane mode *link to bug*
** MP4/3GP *link to bug*
** Can't access emergency dialer *link to bug*
** Scrolling issue on the time picker with b2gdesktop *link to bug*
* Is blocked by design issues in b2g desktop:
* Is blocked by design issues in b2g desktop:
** Different FTUs on b2gdesktop
** Different FTUs on b2gdesktop
** Can't set the time on b2gdesktop without changing the time on the computer
** Can't set the time on b2gdesktop without changing the time on the computer
** Some settings are hidden on b2gdesktop
** Some settings are hidden on b2g desktop

Latest revision as of 17:37, 17 November 2015

In order to make a clearer distinction between the Integration suite and the End-to-end one, here is a list which helps to define where a test belongs depending on its nature.

General considerations

Should be in Integration suite rather than in the End-to-end one:

  • Tests that don't represent a whole user story, such as boundary/edge cases, negative cases, and highly specific or unusual bug verifications.
  • Tests that perform small, isolated checks of the UI (such as a test that only verifies the sort order of a table)
  • Tests that use mocks to simulate a portion of the system
  • Tests that validate binding between the UI state and back end state
  • Tests that verify something which is invisible for an end-user (example: "Is the gaia setting 'ril.data.enabled' correctly set?"). An exception could be made for an End-to-end test if this verification is done during the setUp/tearDown phases.

Should not be in Integration suite:

  • Tests that are flaky by design (networking tests for example)
  • Tests that check a device specific issue.
  • Tests that check the behavior with an external service without being mocked (like a test which interacts with the actual Gmail).
  • Tests that takes too long to be executed.

Firefox OS current specific considerations

A test cannot currently be in the Integration suite, if the test:

  • relies on hardware which can't be currently mocked (like Camera, SIM/DSDS, FM antenna)
  • depends on external website/service which can't be mocked (as Carrier network, Firefox Account, some Email providers)
  • Is blocked by a bug in b2g desktop, (for instance: MP4/3GP not recognized, emergency dialer cannot be open)
  • Is blocked by design issues in b2g desktop:
    • Different FTUs on b2gdesktop
    • Can't set the time on b2gdesktop without changing the time on the computer
    • Some settings are hidden on b2g desktop
Retrieved from "https://wiki.allizom.org/index.php?title=B2G/QA/Automation/UI/Integration_suite_criteria&oldid=1106057"