TestEngineering/Web/Automation/Gaia UI Testing: Difference between revisions

5,774 bytes added , 26 August 2016

m

StuartPhilp moved page QA/Execution/Web Testing/Automation/Gaia UI Testing to TestEngineering/Web/Automation/Gaia UI Testing: reorganizing wiki under Test Engineering

StuartPhilp

81

edits

@@ Line 1: / Line 1: @@
 = Summary =
 This is the home for all documentation related to the Gaia UI tests written and maintained by the Web QA Team.
+= Task List =
+Please see the [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing/Task_List|Gaia UI Testing Task List]] for a list of daily tasks, in order of priority.
 = Running the Tests =
+Please see [https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/gaia-ui-tests Gaia UI Tests Guidelines] on MDN for information on running and writing Gaia UI tests.
 == Running on Desktop B2G ==
-Either add documentation for this here, or link to existing documentation, which I believe is currently in the Github repo.
+Please see [https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Run_Tests#Testing_on_Desktop_B2G Testing on Desktop B2G] on MDN.
 == Running on Device ==
-Either add documentation for this here, or link to existing documentation, which I believe is currently in the Github repo.
+Please see [https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Run_Tests#Testing_on_Devices Testing on Devices] on MDN.
 = Monitoring Jenkins Jobs For Failures and XPasses =
@@ Line 34: / Line 40: @@
 ### You can use the links above to view the latest results of the job. This will give you a quick summary of which tests are currently failing. Looking at the ''Age'' of the failure will give you an idea of whether this is a new failure, or one that has been happening for awhile. Also, failures with an ''Age'' of 1 often indicate intermittent failures.
 ### For each failure that you find:
+#### If the failure details say: '''Error Message''': test failure, '''Stacktrace''': N/A, then the failure is likely actually an xpass. Keep this in mind when going through the next steps.
 #### Search for a bug in Bugzilla that has been opened for the failure. Every failure that needs to be addressed must have a bug opened for it. If you find an open bug for a failure, examine it:
 ##### If it has a pull request attached to it which is waiting to be reviewed, review it. A pull request that is waiting to be reviewed is one that has no negative reviews and less than two positive reviews.
@@ Line 39: / Line 46: @@
 #### If you do not find an open bug for the failure, open one and include as much detail about the failure as possible.
 ##### If you have time to address the failure, assign it to yourself and update its status to ''Assigned''.
-## Review xpasses.
+## Address xpasses.
-### You can also use the links above to view the latest results of the job, but this summary will not show you which jobs are xpassing. To do that you need to view the HTML report which is accessed via a link in the left nav. On the report, look for the summary near the top and look for the text that says 'x unexpected passes'. If 'x' is greater than 0, this build contained xpasses. You will need to look through the HTML report to find the tests that passed unexpectedly.
+### For each xpass that you found during the Review failures step:
-### For each xpass that you find:
 #### Run the test locally to see if it also passes for you.
 ##### If the test passes locally for you and has been passing on CI for more than one consecutive run, then the test should be un-xfailed:
@@ Line 51: / Line 57: @@
 ###### If no bug exists to un-xfail the test, open one.
 ###### Create a pull request to un-xfail the test and attach it to the un-xfail bug mentioned above.
+= Completing Requested Reviews =
+Your second priority, when supporting Gaia UI Testing, is to complete any reviews that have been requested of you. Reviews generally mean reviewing a pull request and Bugzilla is your friend in finding these. You can see all of your requested reviews on your [https://bugzilla.mozilla.org/page.cgi?id=mydashboard.html Buzilla Dashboard] in the section labelled ''Flags Requested of You''.
+Each bug that has you listed as a reviewer with a ''?'' flag will appear on this list. Here is a recommended workflow for each pull request you review. Note that this uses the ''master'' branch as an example.
+# Make sure you are on the correct branch and that it is up to date:
+## <code>git checkout master && git pull upstream master</code>
+# Create a new branch for the review:
+## <code> git checkout -b branch_name</code>
+# Pull the changes for the pull request from the requester's branch:
+## <code> git pull url_of_requesters_repo branch_name</code>
+### Note that you may find it useful to add remotes for any repos for which you are frequently asked to perform reviews. You can do this using <code>git remote add remote_name repo_url</code>, and you can then use <code> git pull remote_name branch_name</code> instead of the above.
+### You should not experience any merge conflicts when doing this pull. If you do, check the pull request. If it indicates that it cannot be merged then add a comment to the pull request that conflicts need to be resolved and flag the attachment on the bug as ''-'' to indicate it did not pass your review.
+#### If the pull request is not marked as being unable to be merged then something went wrong with your pull. Perhaps your local branch was not up to date. Start the process again at #1 above.
+# Run the affected test(s) against the branch in question to ensure that they all pass.
+## Note that there may be several tests affected which may not be included in the pull request. For example, if a locator or method in an app object changes, then any tests that make use of that app object property or method would need to be tested.
+## If the tests do not pass:
+### Check to see if any are xfailed and note these. They are not expected to pass, so they should not affect your review.
+### For any that are not xfailed, add a comment to the pull request to indicate which tests failed. Be sure to include the full stack trace of the failure(s) in your comments. Then flag the attachment on the bug as ''-'' to indicate it did not pass your review.
+## Assuming that the tests are passing, continue to the next step.
+# Review the code in the pull request:
+## Check against the expected style of the gaia-ui-tests.
+## Check for pep8 errors.
+## If any problems are found, add comments to the pull request and flag the attachment on the bug as ''-'' to indicate it did not pass your review.
+## If you got this far and there are no problems with the pull request, flag the attachment on the bug as ''+'' to indicate it passed your review.
+# Merge the pull request:
+## If you are the second reviewer to ''r+'' the pull request then you may merge it. See merging pull requests for details.
+== Merging Pull Requests ==
+Before merging a pull request, please ensure that:
+# The attachment to the bug has at least two positive (''r+'') reviews.
+# The Travis build of the pull request has passed.
+## If the Travis build failed check it to see if the ''gaia_ui_tests'' step failed, and if it did whether any of the affected tests failed. Although it is preferred that no pull request be merged with a failing Travis build, if you are confident that the Travis failure had nothing to do with the changes in the pull request you may merge it, at your own discretion.
+## You may also request a rerun of the Travis build, which you do via the [https://travis-ci.org/mozilla-b2g/gaia/pull_requests Travis UI], if you would prefer to see it pass before merging.
+When it is time to merge, do the following:
+# Visit the pull request and click the big green ''Merge pull request'' button.
+# Go to the bug and ensure that all reviews are cleared.
+# Add a comment to the bug with a link to the merge commit, indicating on which branch the patch landed.
+# If this was a merge to master, consider whether the patch should be uplifted. Follow the procedure in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#After_Changes_Have_Been_Committed_To_Master After|Changes Have Been Committed To Master]].
+# If the bug is finished (e.g., no uplifts are needed), mark it as Resolved - Fixed.
+= Dealing with Travis and TBPL Failures =
+Currently, we do not monitor Travis or TBPL for failures, but we are sometimes informed of them and asked to investigate. When investigating failures on Travis or TBPL, be sure to run your tests against B2G Desktop, not a device, as that is how the tests are run in those environments.
+You can find the executables for B2G desktop at:
+* [https://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/ Master]
+* [https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ v1.2] - look for the ones prefixed with mozilla-b2g26_v1_2, e.g. [https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-b2g26_v1_2-macosx64_gecko/ mozilla-b2g26_v1_2-macosx64_gecko]
+Running like Travis locally.
+There are two steps to running tests like Travis locally; install (downloads Gecko and builds Gaia) and run tests.
+The tests must be run from the gaia/ root directory.
+When running locally we need to be aware of 3 things:
+# The travis install script will not download a new Gecko if one already exists. The solution is to "rm -rf b2g" and "rm -rf profile" to remove the artifacts that the install script creates
+# The install script will install the Python packages at global level. The solution is to remove "sudo" from line 5 of the install script.
+# I forgot wot this one woz but for some reason in my mind I thought there were 3
+The install script is run by:
+> tests/travis_ci/gaia-ui-tests/install
+To run the tests, modify the script file to include the command line options you need. All normal command line options are valid.
+> tests/travis_ci/gaia-ui-tests/script
 = Keeping Branches in Sync with Master =
@@ Line 79: / Line 156: @@
 ## If the commit should be uplifted (or you are unsure, as above), either:
 ### Flag it for uplifting by setting the ''status-b2g-v1.2'' flag to ''affected'', or
-### Complete the uplift yourself. This is preferable to flagging it for someone else to complete. See the section below about [https://wiki.mozilla.org/QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Uplifting_Commits Uplifting Commits].
+### Complete the uplift yourself. This is preferable to flagging it for someone else to complete. See the section below about [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Uplifting_Commits|Uplifting Commits]].
 ### In either case, the bug should not be closed.
 ## If you determine that the commit does not need to be uplifted, flag it as such by setting the ''status-b2g-v1.2'' flag to ''unaffected''. The bug can be closed.
@@ Line 93: / Line 170: @@
 # Cherry pick the commit that was made to master. You can find the commit hash by looking at the Github pull request:
 ## <code> git cherry-pick commit-hash</code>
+## Please note: Use the commit hash of the original commit of code (the one submitted by the pull request author) as opposed to the commit hash of the merge commit (the one created by the person who merged the pull request). For example, for this [https://github.com/mozilla-b2g/gaia/pull/13644 pull request], you would choose commit hash ''a9798fc'', which is the hash of Bob's original commit, not commit hash  ''42de07a'', which is the commit hash of Zac's merge commit.
 ## The cherry pick may or may not complete without conflicts. If there are conflicts, resolve them manually.
-## Please note: Use the commit hash of the original commit of code (the one submitted by the pull request author) as opposed to the commit hash of the merge commit (the one created by the person who merged the pull request). For example, for this [https://github.com/mozilla-b2g/gaia/pull/13644 | pull request], you would choose commit hash ''a9798fc'', which is the hash of Bob's original commit, not commit hash  ''42de07a'', which is the commit hash of Zac's merge commit.
+### If there were conflicts, commit your changes to your branch:
+#### <code>git commit -a</code>
+#### The commit message should be pre-populated with the commit message from when the changes were committed to master. Leave this commit message intact. You may add a second line to the message to indicate that this is an uplift to the branch of interest.
 # Run the affected test(s) against v1.2 to ensure that they still pass.
 ## Note that there may be several tests affected which may not be included in the commit. For example, if a locator or method in an app object changes, then any tests that make use of that app object property or method would need to be tested.
 ## If the tests do not pass, try to determine why and whether or not they can be made to pass by making other changes. If you are unable to get the tests to pass on v1.2, and it isn't because they are attempting to test features that do not exist on v1.2, then make a comment in the bug about this, with the full details of the failures, and leave the bug flagged as ''status-b2g-v1.2: affected''.
 ## Assuming that the tests are passing, continue to the next step.
-# Commit your changes to your branch:
-## <code>git commit -a</code>
-## The commit message should be pre-populated with the commit message from when the changes were committed to master. Leave this commit message intact. You may add a second line to the message to indicate that this is an uplift to the branch of interest.
 # Push your changes to your Github repo:
 ## <code>git push origin branch_name</code>
-# When the pull request for the uplift has been merged, set the ''status-b2g-v1.2'' flag to ''fixed'' and close the bug.
 # Go to Github and issue a pull request against your branch.
 ## Make sure the pull request is being issued against the correct upstream branch (e.g., v1.2). This will likely not be the default so you will have to change it manually.
@@ Line 111: / Line 187: @@
 ## This can also be done by pasting the url to the Github pull request into an attachment to the bug using 'paste text as attachment'.
 ## Flag the attachment for review and add a comment that this patch is to uplift to the branch of interest.
+# When the pull request for the uplift has been merged, set the ''status-b2g-v1.2'' flag to ''fixed'' and close the bug.
 == Reviewing and Merging Uplifts ==
@@ Line 116: / Line 193: @@
 Often a pull request which is an uplift from master to a branch of interest will contain code that is identical to that which was committed to master. That code has already been reviewed for correctness and style, so it may not be necessary to review the code itself as carefully. You will need to determine whether that is the case yourself. You could do that by comparing the files changed, after applying the commit in the pull request, with the same files in the master branch.
-Regardless of the decision made above, before merging the code:
+Regardless of the decision made above, follow the procedure in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Merging_Pull_Requests|Merging Pull Requests]]
-# Any affected tests should be run against the branch in question.
-# A passing Travis build of the pull request should have completed.
-## If the Travis build failed check it to see if the ''gaia_ui_tests'' step failed, and if it did whether any of the affected tests failed. Although it is preferred that no pull request be merged with a failing Travis build, if you are confident that the Travis failure had nothing to do with the changes in your pull request you may merge it, at your own discretion.
-## You may also request a rerun of the Travis build, which you do via the [https://travis-ci.org/mozilla-b2g/gaia/pull_requests Travis UI], if you would prefer to see it pass before merging.
 == Finding Bugs That Require Uplifts ==
-All bugs that require uplifts should be flagged as ''status-b2g-v1.2: affected'' (or whichever flag corresponds to the branch of interest). You can execute a Bugzilla query such as [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=gaia-ui-test-open-v1.2-affected&sharer_id=440009&list_id=8605601 this one], to view all such bugs and then assign yourself to any that you plan to complete.
+All bugs that require uplifts should be flagged as ''status-b2g-v1.2: affected'' (or whichever flag corresponds to the branch of interest). You can execute a Bugzilla query such as [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=gaia-ui-test-open-v1.2-affected-take2&sharer_id=440009&list_id=8786182 this one], to view all such bugs and then assign yourself to any that you plan to complete.
 == Keeping On Top of Changes ==
@@ Line 131: / Line 204: @@
 # As mentioned above, if you can complete an uplift yourself, by creating a pull request, rather than simply flagging a bug as affected, that is preferable.
-# Monitor bugs that are flagged as ''status-b2g-v1.2: affected'' as described in [https://wiki.mozilla.org/QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Finding_Bugs_That_Require_Uplifts Finding Bugs That Require Uplifts].
+# Monitor bugs that are flagged as ''status-b2g-v1.2: affected'' as described in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Finding_Bugs_That_Require_Uplifts|Finding Bugs That Require Uplifts]].
 # Complete a manual audit of commits that have landed in master but not v1.2. This is something that should be done once or twice per month.
 ## Generate a list of all the commits that have landed in master but not v1.2 using this command: <code>git show master ^v1.2 --no-merges -- tests/python/gaia-ui-tests</code>.

TestEngineering/Web/Automation/Gaia UI Testing: Difference between revisions

TestEngineering/Web/Automation/Gaia UI Testing (view source)

Revision as of 20:24, 26 August 2016