81
edits
TestEngineering/Web/Automation/Gaia UI Testing: Difference between revisions
TestEngineering/Web/Automation/Gaia UI Testing (view source)
Revision as of 20:24, 26 August 2016
, 26 August 2016StuartPhilp moved page QA/Execution/Web Testing/Automation/Gaia UI Testing to TestEngineering/Web/Automation/Gaia UI Testing: reorganizing wiki under Test Engineering
StuartPhilp (talk | contribs) m (StuartPhilp moved page QA/Execution/Web Testing/Automation/Gaia UI Testing to TestEngineering/Web/Automation/Gaia UI Testing: reorganizing wiki under Test Engineering) |
|||
| (16 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
= Summary = | = Summary = | ||
This is the home for all documentation related to the Gaia UI tests written and maintained by the Web QA Team. | This is the home for all documentation related to the Gaia UI tests written and maintained by the Web QA Team. | ||
= Task List = | |||
Please see the [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing/Task_List|Gaia UI Testing Task List]] for a list of daily tasks, in order of priority. | |||
= Running the Tests = | = Running the Tests = | ||
Please see [https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/gaia-ui-tests Gaia UI Tests Guidelines] on MDN for information on running and writing Gaia UI tests. | |||
== Running on Desktop B2G == | == Running on Desktop B2G == | ||
Please see [https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Run_Tests#Testing_on_Desktop_B2G Testing on Desktop B2G] on MDN. | |||
== Running on Device == | == Running on Device == | ||
Please see [https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Run_Tests#Testing_on_Devices Testing on Devices] on MDN. | |||
= Monitoring Jenkins Jobs For Failures and XPasses = | = Monitoring Jenkins Jobs For Failures and XPasses = | ||
| Line 80: | Line 86: | ||
# Merge the pull request: | # Merge the pull request: | ||
## If you are the second reviewer to ''r+'' the pull request then you may merge it. See merging pull requests for details. | ## If you are the second reviewer to ''r+'' the pull request then you may merge it. See merging pull requests for details. | ||
== Merging Pull Requests == | |||
Before merging a pull request, please ensure that: | |||
# The attachment to the bug has at least two positive (''r+'') reviews. | |||
# The Travis build of the pull request has passed. | |||
## If the Travis build failed check it to see if the ''gaia_ui_tests'' step failed, and if it did whether any of the affected tests failed. Although it is preferred that no pull request be merged with a failing Travis build, if you are confident that the Travis failure had nothing to do with the changes in the pull request you may merge it, at your own discretion. | |||
## You may also request a rerun of the Travis build, which you do via the [https://travis-ci.org/mozilla-b2g/gaia/pull_requests Travis UI], if you would prefer to see it pass before merging. | |||
When it is time to merge, do the following: | |||
# Visit the pull request and click the big green ''Merge pull request'' button. | |||
# Go to the bug and ensure that all reviews are cleared. | |||
# Add a comment to the bug with a link to the merge commit, indicating on which branch the patch landed. | |||
# If this was a merge to master, consider whether the patch should be uplifted. Follow the procedure in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#After_Changes_Have_Been_Committed_To_Master After|Changes Have Been Committed To Master]]. | |||
# If the bug is finished (e.g., no uplifts are needed), mark it as Resolved - Fixed. | |||
= Dealing with Travis and TBPL Failures = | |||
Currently, we do not monitor Travis or TBPL for failures, but we are sometimes informed of them and asked to investigate. When investigating failures on Travis or TBPL, be sure to run your tests against B2G Desktop, not a device, as that is how the tests are run in those environments. | |||
You can find the executables for B2G desktop at: | |||
* [https://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/ Master] | |||
* [https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ v1.2] - look for the ones prefixed with mozilla-b2g26_v1_2, e.g. [https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-b2g26_v1_2-macosx64_gecko/ mozilla-b2g26_v1_2-macosx64_gecko] | |||
Running like Travis locally. | |||
There are two steps to running tests like Travis locally; install (downloads Gecko and builds Gaia) and run tests. | |||
The tests must be run from the gaia/ root directory. | |||
When running locally we need to be aware of 3 things: | |||
# The travis install script will not download a new Gecko if one already exists. The solution is to "rm -rf b2g" and "rm -rf profile" to remove the artifacts that the install script creates | |||
# The install script will install the Python packages at global level. The solution is to remove "sudo" from line 5 of the install script. | |||
# I forgot wot this one woz but for some reason in my mind I thought there were 3 | |||
The install script is run by: | |||
> tests/travis_ci/gaia-ui-tests/install | |||
To run the tests, modify the script file to include the command line options you need. All normal command line options are valid. | |||
> tests/travis_ci/gaia-ui-tests/script | |||
= Keeping Branches in Sync with Master = | = Keeping Branches in Sync with Master = | ||
| Line 108: | Line 156: | ||
## If the commit should be uplifted (or you are unsure, as above), either: | ## If the commit should be uplifted (or you are unsure, as above), either: | ||
### Flag it for uplifting by setting the ''status-b2g-v1.2'' flag to ''affected'', or | ### Flag it for uplifting by setting the ''status-b2g-v1.2'' flag to ''affected'', or | ||
### Complete the uplift yourself. This is preferable to flagging it for someone else to complete. See the section below about [ | ### Complete the uplift yourself. This is preferable to flagging it for someone else to complete. See the section below about [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Uplifting_Commits|Uplifting Commits]]. | ||
### In either case, the bug should not be closed. | ### In either case, the bug should not be closed. | ||
## If you determine that the commit does not need to be uplifted, flag it as such by setting the ''status-b2g-v1.2'' flag to ''unaffected''. The bug can be closed. | ## If you determine that the commit does not need to be uplifted, flag it as such by setting the ''status-b2g-v1.2'' flag to ''unaffected''. The bug can be closed. | ||
| Line 122: | Line 170: | ||
# Cherry pick the commit that was made to master. You can find the commit hash by looking at the Github pull request: | # Cherry pick the commit that was made to master. You can find the commit hash by looking at the Github pull request: | ||
## <code> git cherry-pick commit-hash</code> | ## <code> git cherry-pick commit-hash</code> | ||
## Please note: Use the commit hash of the original commit of code (the one submitted by the pull request author) as opposed to the commit hash of the merge commit (the one created by the person who merged the pull request). For example, for this [https://github.com/mozilla-b2g/gaia/pull/13644 pull request], you would choose commit hash ''a9798fc'', which is the hash of Bob's original commit, not commit hash ''42de07a'', which is the commit hash of Zac's merge commit. | |||
## The cherry pick may or may not complete without conflicts. If there are conflicts, resolve them manually. | ## The cherry pick may or may not complete without conflicts. If there are conflicts, resolve them manually. | ||
## | ### If there were conflicts, commit your changes to your branch: | ||
#### <code>git commit -a</code> | |||
#### The commit message should be pre-populated with the commit message from when the changes were committed to master. Leave this commit message intact. You may add a second line to the message to indicate that this is an uplift to the branch of interest. | |||
# Run the affected test(s) against v1.2 to ensure that they still pass. | # Run the affected test(s) against v1.2 to ensure that they still pass. | ||
## Note that there may be several tests affected which may not be included in the commit. For example, if a locator or method in an app object changes, then any tests that make use of that app object property or method would need to be tested. | ## Note that there may be several tests affected which may not be included in the commit. For example, if a locator or method in an app object changes, then any tests that make use of that app object property or method would need to be tested. | ||
## If the tests do not pass, try to determine why and whether or not they can be made to pass by making other changes. If you are unable to get the tests to pass on v1.2, and it isn't because they are attempting to test features that do not exist on v1.2, then make a comment in the bug about this, with the full details of the failures, and leave the bug flagged as ''status-b2g-v1.2: affected''. | ## If the tests do not pass, try to determine why and whether or not they can be made to pass by making other changes. If you are unable to get the tests to pass on v1.2, and it isn't because they are attempting to test features that do not exist on v1.2, then make a comment in the bug about this, with the full details of the failures, and leave the bug flagged as ''status-b2g-v1.2: affected''. | ||
## Assuming that the tests are passing, continue to the next step. | ## Assuming that the tests are passing, continue to the next step. | ||
# Push your changes to your Github repo: | # Push your changes to your Github repo: | ||
## <code>git push origin branch_name</code> | ## <code>git push origin branch_name</code> | ||
# Go to Github and issue a pull request against your branch. | # Go to Github and issue a pull request against your branch. | ||
## Make sure the pull request is being issued against the correct upstream branch (e.g., v1.2). This will likely not be the default so you will have to change it manually. | ## Make sure the pull request is being issued against the correct upstream branch (e.g., v1.2). This will likely not be the default so you will have to change it manually. | ||
| Line 140: | Line 187: | ||
## This can also be done by pasting the url to the Github pull request into an attachment to the bug using 'paste text as attachment'. | ## This can also be done by pasting the url to the Github pull request into an attachment to the bug using 'paste text as attachment'. | ||
## Flag the attachment for review and add a comment that this patch is to uplift to the branch of interest. | ## Flag the attachment for review and add a comment that this patch is to uplift to the branch of interest. | ||
# When the pull request for the uplift has been merged, set the ''status-b2g-v1.2'' flag to ''fixed'' and close the bug. | |||
== Reviewing and Merging Uplifts == | == Reviewing and Merging Uplifts == | ||
| Line 145: | Line 193: | ||
Often a pull request which is an uplift from master to a branch of interest will contain code that is identical to that which was committed to master. That code has already been reviewed for correctness and style, so it may not be necessary to review the code itself as carefully. You will need to determine whether that is the case yourself. You could do that by comparing the files changed, after applying the commit in the pull request, with the same files in the master branch. | Often a pull request which is an uplift from master to a branch of interest will contain code that is identical to that which was committed to master. That code has already been reviewed for correctness and style, so it may not be necessary to review the code itself as carefully. You will need to determine whether that is the case yourself. You could do that by comparing the files changed, after applying the commit in the pull request, with the same files in the master branch. | ||
Regardless of the decision made above, | Regardless of the decision made above, follow the procedure in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Merging_Pull_Requests|Merging Pull Requests]] | ||
== Finding Bugs That Require Uplifts == | == Finding Bugs That Require Uplifts == | ||
All bugs that require uplifts should be flagged as ''status-b2g-v1.2: affected'' (or whichever flag corresponds to the branch of interest). You can execute a Bugzilla query such as [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=gaia-ui-test-open-v1.2-affected&sharer_id=440009&list_id= | All bugs that require uplifts should be flagged as ''status-b2g-v1.2: affected'' (or whichever flag corresponds to the branch of interest). You can execute a Bugzilla query such as [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=gaia-ui-test-open-v1.2-affected-take2&sharer_id=440009&list_id=8786182 this one], to view all such bugs and then assign yourself to any that you plan to complete. | ||
== Keeping On Top of Changes == | == Keeping On Top of Changes == | ||
| Line 160: | Line 204: | ||
# As mentioned above, if you can complete an uplift yourself, by creating a pull request, rather than simply flagging a bug as affected, that is preferable. | # As mentioned above, if you can complete an uplift yourself, by creating a pull request, rather than simply flagging a bug as affected, that is preferable. | ||
# Monitor bugs that are flagged as ''status-b2g-v1.2: affected'' as described in [ | # Monitor bugs that are flagged as ''status-b2g-v1.2: affected'' as described in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Finding_Bugs_That_Require_Uplifts|Finding Bugs That Require Uplifts]]. | ||
# Complete a manual audit of commits that have landed in master but not v1.2. This is something that should be done once or twice per month. | # Complete a manual audit of commits that have landed in master but not v1.2. This is something that should be done once or twice per month. | ||
## Generate a list of all the commits that have landed in master but not v1.2 using this command: <code>git show master ^v1.2 --no-merges -- tests/python/gaia-ui-tests</code>. | ## Generate a list of all the commits that have landed in master but not v1.2 using this command: <code>git show master ^v1.2 --no-merges -- tests/python/gaia-ui-tests</code>. | ||