B2G/QA/Automation/UI/Strategy/Develop Gaia End to end: Difference between revisions

From MozillaWiki
< B2G‎ | QA‎ | Automation‎ | UI‎ | Strategy
Jump to navigation Jump to search
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
=Objective=
=Objective=


Create a harness on top of Marionette JS that executes and controls a device with the level of care and confidence appropriate to UATs, as established via Gaia UI Tests in Python.
Create a test suite on top of Marionette JS that executes and controls a device with the level of care and confidence appropriate to acceptance testing, as established via Gaia UI Tests in Python.


=Challenges Addressed=
=Challenges Addressed=
Line 9: Line 9:
=The Problem=
=The Problem=


UATs are currently written in Python on top of the Gaia UI Test harness. While this has worked well for us, the organization in general is moving towards JavaScript.  
Acceptance tests are currently written in Python on top of the Gaia UI Test harness. While this has worked well for us, the organization in general is moving towards JavaScript.  


This heavily limits our ability to get development to cooperate in reviewing or making changes to our UATs. While QA does own these, some assistance is helpful and Python is a blocker.
This heavily limits our ability to get development to cooperate in reviewing or making changes to our acceptance automation. While QA does own these, some assistance is helpful and Python is a blocker.


In addition, Views (page object equivalents) will be created first in JavaScript to handle CI testing. If we do not align, we will have to reimplement the same basic classes in Python. Further, by using common view objects, developers will effectively maintain the UATs on the way to keeping CI tests green.
In addition, Views (page object equivalents) will be created first in JavaScript to handle Gaia Integration testing. If we do not align, we will have to reimplement the same basic classes in Python. Further, by using common view objects, developers will effectively maintain the Gaia Acceptance tests on the way to keeping Gaia Integration tests green.


=The Solution=
=The Solution=
Line 23: Line 23:
Q1:
Q1:


* Transition schedule/gates [Geo]
* {{done|Transition schedule/gates [Geo]}}
* Documentation of Gaia UI Test harness behavior [Geo]
** https://etherpad.mozilla.org/marionette-js-transition
* Gaia UAT v1 implementation [Geo]
* {{missed|Documentation of Gaia UI Test harness behavior [Geo]}}
* Review and improvement if necessary of View pattern [John]
* {{missed|Gaia Acceptance Proof of Concept implementation [Geo]}}
* Establish JS Gaia UAT backlog with existing Python UATs to port [Johan]
* {{missed|Review and improvement if necessary of View pattern [John]}}
** Partial, view pattern PoC established and being used, but not finalized.
* {{done|Establish Gaia Acceptance backlog with existing Python UI tests to port [Johan]}}[https://bugzilla.mozilla.org/show_bug.cgi?id=1146909]
 
Ongoing:
 
* Help fix Marionette JS on-device when available [Martijn]
* Conduct Q1 Pilot Program (Gaia Integration runs gij on-device) [Martijn]
* Conduct Q1 Pilot Program (Gaia Integration runs gij on-device) [Martijn]
* Help fix Marionette JS on-device when available [Martijn]
* If viable in Q1, start porting smoketests to Gaia Acceptance
* If viable in Q1, start porting smoketests to Gaia UAT


Q2:
Q2:
Line 38: Line 43:
=Risks=
=Risks=


* UATs must be able to work on-device. Marionette JS is not currently stable on device.
* Acceptance tests must be able to work on-device. Marionette JS is not currently stable on device.
* The gap between Python Gaia UI Test and Marionette JS is not fully known yet.
* The gap between Python Gaia UI Test and Marionette JS is not fully known yet.

Latest revision as of 17:51, 17 November 2015

Objective

Create a test suite on top of Marionette JS that executes and controls a device with the level of care and confidence appropriate to acceptance testing, as established via Gaia UI Tests in Python.

Challenges Addressed

  • UI tests have had no developer support, largely because of Python implementation

The Problem

Acceptance tests are currently written in Python on top of the Gaia UI Test harness. While this has worked well for us, the organization in general is moving towards JavaScript.

This heavily limits our ability to get development to cooperate in reviewing or making changes to our acceptance automation. While QA does own these, some assistance is helpful and Python is a blocker.

In addition, Views (page object equivalents) will be created first in JavaScript to handle Gaia Integration testing. If we do not align, we will have to reimplement the same basic classes in Python. Further, by using common view objects, developers will effectively maintain the Gaia Acceptance tests on the way to keeping Gaia Integration tests green.

The Solution

Analyze Gaia UI Test on Python and implement any missing pieces on top of Marionette JS and Mocha, as appropriate. Port existing tests to the new harness.

Timeline

Q1:

  • [DONE] Transition schedule/gates [Geo]
  • [MISSED] Documentation of Gaia UI Test harness behavior [Geo]
  • [MISSED] Gaia Acceptance Proof of Concept implementation [Geo]
  • [MISSED] Review and improvement if necessary of View pattern [John]
    • Partial, view pattern PoC established and being used, but not finalized.
  • [DONE] Establish Gaia Acceptance backlog with existing Python UI tests to port [Johan][1]

Ongoing:

  • Help fix Marionette JS on-device when available [Martijn]
  • Conduct Q1 Pilot Program (Gaia Integration runs gij on-device) [Martijn]
  • If viable in Q1, start porting smoketests to Gaia Acceptance

Q2:

  • Port existing tests not addressed in Q1

Risks

  • Acceptance tests must be able to work on-device. Marionette JS is not currently stable on device.
  • The gap between Python Gaia UI Test and Marionette JS is not fully known yet.
Retrieved from "https://wiki.allizom.org/index.php?title=B2G/QA/Automation/UI/Strategy/Develop_Gaia_End_to_end&oldid=1106067"