B2G/QA/Automation/UI/Strategy/Assist with Gaia Integration tests: Difference between revisions
| (12 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
=Objective= | =Objective= | ||
Increase stability of product and further relationship with development by helping them expand the | Increase stability of product and further relationship with development by helping them expand the suite of Gaia Integration tests. | ||
=Challenges Addressed= | =Challenges Addressed= | ||
| Line 7: | Line 7: | ||
* Development and QA have different approaches and needs from UI testing | * Development and QA have different approaches and needs from UI testing | ||
* Test results are hidden in Jenkins | * Test results are hidden in Jenkins | ||
* Test results only looked at a limited number of times per day | * Test results only looked at a limited number of times per day | ||
* The product is too unstable for acceptance testing during development phases | |||
=The Problem= | =The Problem= | ||
There aren't enough Gaia Integration tests. Builds are constantly unstable. Also, since Acceptance tests primarily benefit QA directly, | There aren't enough Gaia Integration tests. Builds are constantly unstable. Also, since Acceptance tests primarily benefit QA directly, the larger organization doesn't always see the benefit they get from us. | ||
=The Solution= | =The Solution= | ||
Add more Gaia Integration tests covering any behavior reasonable. Don't worry about whether the tests are sufficient for Acceptance. Offer help to functional teams whenever possible. | Add more Gaia Integration tests covering any behavior reasonable. Don't worry about whether the tests are sufficient for Acceptance. Offer help to functional teams whenever possible. Make sure our contributions are visible. | ||
=Timeline= | =Timeline= | ||
| Line 21: | Line 22: | ||
Q1 | Q1 | ||
* Criteria document for picking Gaia Integration tests to automate [Johan] | * {{done|Criteria document for picking Gaia Integration tests to automate [Johan]}} [https://bugzilla.mozilla.org/show_bug.cgi?id=1136779] | ||
* Analysis of Acceptance tests for additional Gaia Integration possibilities [Johan] | * {{done|Analysis of Acceptance tests for additional Gaia Integration possibilities [Johan]}} [https://bugzilla.mozilla.org/show_bug.cgi?id=1139921] | ||
* Propose/add tests to functional team backlogs [John] | * {{missed|Propose/add tests to functional team backlogs [John]}} | ||
Ongoing: | |||
* Work with functional teams to implement agreed-upon tests | * Work with functional teams to implement agreed-upon tests | ||
* After verifying via functional team QA leads, help add more tests | * After verifying via functional team QA leads, help add more tests | ||
| Line 29: | Line 33: | ||
=Risks= | =Risks= | ||
QA can't handle the weight of ownership of all the Gaia Integration tests for the entire product. They should be written by the domain experts anyway, and they need to be maintained with the code in the same checkin. | |||
That means functional teams need to continue to own their own tests, with QA assisting as part of their functional team membership. Part of ownership is determining how many tests can be maintained given available resources and scoping coverage accordingly. | |||
Because of this, we need to make sure we've verified functional teams want the tests before we add them. | |||
Latest revision as of 16:55, 9 April 2015
Objective
Increase stability of product and further relationship with development by helping them expand the suite of Gaia Integration tests.
Challenges Addressed
- Development and QA have different approaches and needs from UI testing
- Test results are hidden in Jenkins
- Test results only looked at a limited number of times per day
- The product is too unstable for acceptance testing during development phases
The Problem
There aren't enough Gaia Integration tests. Builds are constantly unstable. Also, since Acceptance tests primarily benefit QA directly, the larger organization doesn't always see the benefit they get from us.
The Solution
Add more Gaia Integration tests covering any behavior reasonable. Don't worry about whether the tests are sufficient for Acceptance. Offer help to functional teams whenever possible. Make sure our contributions are visible.
Timeline
Q1
- [DONE] Criteria document for picking Gaia Integration tests to automate [Johan] [1]
- [DONE] Analysis of Acceptance tests for additional Gaia Integration possibilities [Johan] [2]
- [MISSED] Propose/add tests to functional team backlogs [John]
Ongoing:
- Work with functional teams to implement agreed-upon tests
- After verifying via functional team QA leads, help add more tests
Risks
QA can't handle the weight of ownership of all the Gaia Integration tests for the entire product. They should be written by the domain experts anyway, and they need to be maintained with the code in the same checkin.
That means functional teams need to continue to own their own tests, with QA assisting as part of their functional team membership. Part of ownership is determining how many tests can be maintained given available resources and scoping coverage accordingly.
Because of this, we need to make sure we've verified functional teams want the tests before we add them.