MozillaWiki

Phabricator/TestPlan

< Phabricator
Revision as of 14:33, 15 November 2017 by Chartjes (talk | contribs)

Getting Started

The following steps are required to get QA up and running for Phabricator testing on Mozilla's staging server.

NOTE: At this time we are using the dev server for QA test runs rather than staging, as the latter is on the VPN, which has been causing some difficulties.

1. Clone the following repositories to the same directory on your machine:

2. Add arc to your path. On OS X and Linux (and the Windows 10 Linux Subsystem), you can add the following to your `~/.bash_profile` (modifying the path appropriately): 

   alias arc='/Users/youruser/path/to/arcanist/bin/arc'

Doing so will allow you to use the arc command from anywhere on your machine.

3. Clone the repo(s) you plan to test on:

4. In the local repository checkout, run: 

   arc install-certificate

Follow the instructions presented in the command line to associate your local machine with the Phabricator instance via an API key.

5. Test away! You can create branches within your local repository checkout, add commits, and send them to Phabricator via arc diff .^. See the Mozilla Phabricator User Documentation for more.

This should be all you need to get going with arc and our staging and dev servers!

Test Plan

These tests will help ensure that our extensions continue to work as expected when Phabricator is upgraded. They are important as Phacility make no guarantees that internal modules/objects/APIs won't change. See the upgrade process for more.

Please update this doc with new tests as features are added/changed and as holes in the plan are discovered.

Signing up is successful

Test Plan

  1. Create a new BMO account.
  2. Make sure to include an irc nick in the Real Name field.
    • E.g. "John Doe [:jdoe]"
  3. Go to Phabricator and login as the new user.
  4. Finish account creation.

Results

  1. After logging into Phabricator, on the "Create a New Account" page,
    1. Username should be prefilled with the irc nick,
    2. Real Name is correct and does not contain [] or any other extraneous characters.
  2. Clicking register approval completes account creation without error.
  3. The account works as expected.

Creating a revision is successful

Test Plan

  1. Make sure you have arc properly setup on your machine and have run arc install-certificate using the Phabricator user you wish to test with as documented above.
  2. Go to BMO and create a new bug as the Bugzilla user that the Phabricator account is connected to. (Or use an existing public bug).
  3. Using the repo listed in the "Getting Started" section above that matches the environment you are testing, make some change to a file.
  4. Run hg commit -A -m 'commit1: New changes
  5. Run arc diff .^
  6. Input something for the title, summary, test plan, and the correct bug id.
  7. Save and exit the file.

Results

  1. arc diff only submitted the 1 newly created commit.
  2. Visit Phabricator and there should be a new revision with the title.
  3. The revision should contain the correct diff of the changes that were made.
  4. The "Bugzilla Bug ID" is correct.
    1. The Bug number is correct.
    2. The link is correct based on the environment (bugzilla.allizom.org for staging, bugzilla-dev.allizom.org for dev).
    • Note: If the bug id is short (like 3, 92, etc, which is common on local test environments) the link will not work but that is fine because it only works on very long bug ids which match what new bugs on production bmo will be using.
  5. Visiting the bug on bugzilla shows an x-phabricator-request attachment.

Updating a revision with an amended commit is successful

Test Plan

  1. hg update to a commit which you previously submitted with arc
  2. Make a change and run hg commit --amend
  3. Run arc diff .^

Results

  1. arc should automatically detect a revision and ask you if you want to update it.
  2. The revision is updated with the new diff on Phabricator:
    1. The History tab should have two entries, Diff 1 and Diff 2.
    2. The Commits tab should have a single entry.
  3. The bug id and other information remains unchanged.

Creating a secure revision is successful

Your Bugzilla user must belong to a security group, e.g. core-security.

Test Plan

  1. Go to bugzilla and create a security bug:
    • Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
  2. Create a new hg commit.
  3. Run arc diff .^.
  4. Enter the title, summary, test plan, and the bug id of the security bug.
  5. Submit the revision.

Results

  1. The diff and information of the revision are as expected.
  2. The revision has a "Custom Policy" attached to it.
  3. The revision added the creator as a subscriber.
  4. The revision is visible to the user who made it.
  5. The revision is visible to users belonging to the security groups of the bug.
  6. The revision is NOT visible to the public without logging in.
  7. The revision is NOT visible to logged in members without the correct permission.
  8. There is an x-phabricator-request attachment on the bug in Bugzilla.

Creating a revision checks the bug id

Test Plan

  1. Create a new hg commit.
  2. Run arc diff .^
  3. Enter information for the title, summary, and test plan.
  4. Enter the bug id as specified for each expected result below.

Results

  1. Entering no bug id fails with a helpful message.
  2. Entering an invalid bug id, e.g "abcd efg" or "$1000", fails.
  3. Entering the id of a bug that does not exist fails.
  4. Entering the id of a bug of a secure revision that the user does not have access to fails.
  5. Entering a valid bug id is successful.

Creating multiple revisions with the same bug id is successful

Test Plan

  1. Create a new hg commit.
  2. Run arc diff .^
  3. Enter information for the title, summary, and test plan.
  4. Enter the bug id.
  5. Repeat and create another revision with the same bug id.

Result

  1. Both revisions are created successfully.
  2. There are 2 corresponding x-phabricator-request attachments on the bug in Bugzilla.

Requesting and leaving a review on a revision is successful

Test Plan

  1. Ensure that you have 2 Phabricator accounts that log in via Bugzilla ready to go.
  2. Create a commit, run arc diff .^.
  3. Input the title, summary, test plan, and bug id of a public bug.
  4. For the "Reviewers" field enter the Phabricator user name of the other account.
  5. Log into Phabricator as the reviewer account.
  6. Add the "Accept Revision" action at the bottom.
  7. Add the "Request Changes" action at the bottom.

Results

  1. The Phabricator attachment on Bugzilla is present.
  2. The Phabricator attachment on the bug should have no r+ flags.

Resigning as a reviewer removes r+

Test Plan

  1. Ensure that you have 2 Phabricator accounts that log in via Bugzilla ready to go.
  2. Create a commit, run arc diff .^.
  3. Input the title, summary, test plan, and bug id of a public bug.
  4. For the "Reviewers" field enter the Phabricator user name of the other account.
  5. Log into Phabricator as the reviewer account.
  6. Add the "Accept Revision" action at the bottom.
  7. Add the "Resign as Reviewer" action at the bottom.

Results

  1. The Phabricator attachment on Bugzilla is present.
  2. The Phabricator attachment on the bug should have no r+ flags.

Plan Changes clears r+s

Test Plan

  1. Ensure that you have 2 Phabricator accounts that log in via Bugzilla ready to go.
  2. Create a commit, run arc diff .^.
  3. Input the title, summary, test plan, and bug id of a public bug.
  4. For the "Reviewers" field enter the Phabricator user name of the other account.
  5. Log into Phabricator as the reviewer account.
  6. Add the "Accept Revision" action at the bottom.
  7. Log into Phabricator as the revision author account.
  8. Add the "Plan Changes" action at the bottom.

Results

  1. The Phabricator attachment on Bugzilla is present.
  2. The Phabricator attachment in Bugzilla does not have any r+ flags.

Abandoning a revision obsoletes the attachment

Test Plan

  1. Create a new bug.
  2. Create a commit and run arc diff .^.
  3. Input the title, summary, test plan, and bug id of the newly created bug.
  4. Go to the newly created revision and perform the "Abandon Revision" action from the "Add Action" dropdown.

Results

  1. The bug has no active attachments listed.
  2. The bug has one obsolete attachment listed, linking to the created revision.

Reclaiming a revision unobsoletes the attachment

Test Plan

  1. In the revision created as part of the "Abandoning a revision obsoletes the attachment" test, perform the "Reclaim revision" action.

Results

  1. The bug's Phabricator attachment is unobsoleted.
Retrieved from "https://wiki.allizom.org/index.php?title=Phabricator/TestPlan&oldid=1184237"