B2G/QA/Automation/UI/Strategy

Team Mandates

• Establish visible subject matter expertise and impact in test and automation
• Own, expand and maintain QA End-to-end automation
• Increase confidence in End-to-end test automation
• Provide expert assistance to improve all phases of UI test coverage
• Catch problems quickly and effectively
• Increase community support for FxOS QA
• Increase quality of and confidence in the product

Challenges

• Automation systems and concerns are poorly documented
• Development and QA have different approaches and needs from UI testing
• UI tests have had no developer support, largely because of Python implementation
• Team has been blocked too much by cross-team dependencies
• Test results are hidden in Jenkins
• Test results have too many spurious failures to sheriff
• Test results only looked at a limited number of times per day
• The product is too unstable for end-to-end testing during development phases
• End-to-end coverage is insufficient to guarantee a quality build
• Community paths are poorly defined and advertised

Plan

• Separate automation concerns for Integration and End-to-end tests

For End-to-end:

• Streamline execution for increased responsiveness with less effort
• Document how we develop and execute End-to-end automation
• Develop Gaia End-to-end, based on Marionette JS
• Increase coverage using Gaia End-to-end

For Integration:

• Assist the functional teams by increasing Gaia Integration coverage
• Port Gaia Integration Python tests to JavaScript for wider ownership

For Community:

• Find and implement new approaches to increasing community involvement