B2G/QA/Automation/UI/Strategy

FxOS QA UI Automation Strategy

Objectives

• Establish visible subject matter expertise in automation
• Own, expand and maintain QA UAT automation
• Increase confidence in both UAT automation and the product
• Provide expert assistance to improve all phases of UI test coverage
• Catch problems quickly and effectively
• Increase community support for FxOS QA
• Increase quality

Challenges

• Automation systems and concerns are poorly documented
• Development and QA have different approaches and needs from UI testing
• UI tests have had no developer support, largely because of Python implementation
• Team has been blocked too much by cross-team dependencies
• Test results are hidden in Jenkins
• Test results have too many spurious failures to sheriff
• Test results only looked at a limited number of times per day
• Community paths poorly defined and advertised

Plan

• Separate automation concerns for Continuous Integration (CI) and User Acceptance (UAT) testing

For UATs:

• Streamline current execution for increased responsiveness
• Document the automation system and process
• Develop a GaiaTest-equivalent harness based on Marionette JS
• Increase the coverage using the Marionette JS GaiaTest harness

For CIs:

• Assist the functional teams by increasing coverage
• Move existing Gaia Integration Python tests to Gaia Integration JS for wider ownership

For Community:

• Find and implement new approaches to increasing community involvement