TestEngineering/Web/Automation/Gaia UI Testing

Summary

This is the home for all documentation related to the Gaia UI tests written and maintained by the Web QA Team.

Running the Tests

Running on Desktop B2G

Either add documentation for this here, or link to existing documentation, which I believe is currently in the Github repo.

Running on Device

Either add documentation for this here, or link to existing documentation, which I believe is currently in the Github repo.

Keeping Branches in Sync with Master

Currently, all development of Gaia UI tests is done on the master branch, with the exception of fixes that are needed specifically for a different branch. It is our goal to keep other branches of interest as closely synced with master as possible. When changes are committed to master, it is important to determine whether those changes should also be uplifted to other branches, and if so then the uplift must be completed.

Current Branches of Interest

• Master - this is the current development branch of Gaia and is priority #1 for failing tests. All new development (e.g., new tests) should be completed in the master branch and then, if necessary, uplifted to other branches of interest.
• v1.2 - this is the next shipping version of Gaia, and is priority #2 for failing tests. Changes should only be committed to v1.2 if they either:
  • land first in master and are deemed necessary for v1.2, or
  • contain fixes that are specific to v1.2

This means that the only current branches of interest are master and v1.2. For this reason the rest of this page will use v1.2 to mean any branch of interest.

After Changes Have Been Committed To Master

The following steps should be taken:

1. Determine whether the change should be uplifted to v1.2. There are a number of considerations for this including:
   1. If it is a new test:
      1. Are the features being tested available in v1.2? If yes, then it should be uplifted, otherwise do not uplift.
   2. If it is a fix to a failing test on master:
      1. Is it a locator or other change that causes the test to fail on master but not on v1.2? If yes, then do not uplift.
      2. Is it a test enhancement, such as an improved wait? If yes, then it should be uplifted.
   3. If, after considering the above, you are unsure as to whether the commit should be uplifted to v1.2, then flag it for uplifting and add a comment that you are not sure if it is needed or not. By flagging it for uplifting you are ensuring that someone will look at it and make that determination.
2. Flag the bug to indicate if it is to be uplifted or not, or complete the uplift yourself:
   1. If the commit should be uplifted (or you are unsure, as above), either:
      1. Flag it for uplifting by setting the status-b2g-v1.2 flag to affected, or
      2. Complete the uplift yourself. This is preferable to flagging it for someone else to complete. See the section below about Uplifting Commits.
      3. In either case, the bug should not be closed.
   2. If you determine that the commit does not need to be uplifted, flag it as such by setting the status-b2g-v1.2 flag to unaffected. The bug can be closed.

Uplifting Commits

Uplifting means to take the changes in a commit made to master, and bring those same changes into the branch of interest. Here is a recommended workflow. Note that this uses the v1.2 branch as an example.

1. Make sure you are on the correct branch and that it is up to date:
   1. git checkout v1.2 && git pull upstream v1.2
2. Create a new branch for the uplift:
   1. git checkout -b branch_name
3. Cherry pick the commit that was made to master. You can find the commit hash by looking at the Github pull request:
   1. git cherry-pick commit-hash
   2. The cherry pick may or may not complete without conflicts. If there are conflicts, resolve them manually.
4. Run the affected test(s) against v1.2 to ensure that they still pass.
   1. Note that there may be several tests affected which may not be included in the commit. For example, if a locator or method in an app object changes, then any tests that make use of that app object property or method would need to be tested.
   2. If the tests do not pass, try to determine why and whether or not they can be made to pass by making other changes. If you are unable to get the tests to pass on v1.2, and it isn't because they are attempting to test features that do not exist on v1.2, then make a comment in the bug about this, with the full details of the failures, and leave the bug flagged as status-b2g-v1.2: affected.
   3. Assuming that the tests are passing, continue to the next step.
5. Commit your changes to your branch:
   1. git commit -a
   2. The commit message should be pre-populated with the commit message from when the changes were committed to master. Leave this commit message intact. You may add a second line to the message to indicate that this is an uplift to the branch of interest.
6. Push your changes to your Github repo:
   1. git push origin branch_name
7. When the pull request for the uplift has been merged, set the status-b2g-v1.2 flag to fixed and close the bug.
8. Go to Github and issue a pull request against your branch.
   1. Make sure the pull request is being issued against the correct upstream branch (e.g., v1.2). This will likely not be the default so you will have to change it manually.
9. Attach the pull request to the bug.
   1. This can be done from the pull request if you have the Github Bugzilla Tweaks add-on installed.
   2. This can also be done by pasting the url to the Github pull request into an attachment to the bug using 'paste text as attachment'.
   3. Flag the attachment for review and add a comment that this patch is to uplift to the branch of interest.

Keeping On Top of Changes

There are some steps that can be taken to try to keep the branches in sync. These include:

1. As mentioned above, if you can complete an uplift yourself, by creating a pull request, rather than simply flagging a bug as affected, that is preferable.
2. Monitor bugs that are flagged as status-b2g-v1.2: affected, and complete the uplifts for those. You can use a shared Bugzilla query that was created for this purpose to view such bugs.
3. Complete an audit of commits that have landed in master but not v1.2.
   1. You can use the command git show master ^v1.2 --no-merges -- tests/python/gaia-ui-tests to show all such commits.