Phabricator/TestPlan

Getting Started

The following steps are required to get QA up and running for Phabricator testing on Mozilla's staging server.

NOTE: We are using the dev server for QA test runs.

1. Clone the following repositories to the same directory on your machine:

   git clone https://github.com/phacility/libphutil
   git clone https://github.com/phacility/arcanist
   git clone https://github.com/mozilla-conduit/arcanist --branch master --single-branch cinnabarc

2. Add vanilla arcanist (bin/arc from your phacility repo checkout) to your path. On OS X and Linux (and the Windows 10 Linux Subsystem), you can add the following to your `~/.bash_profile` (modifying the path appropriately):

   alias arc='/Users/youruser/path/to/arcanist/bin/arc'

Doing so will allow you to use the arc command from anywhere on your machine.

3. Install git-cinnabar:

   git clone https://github.com/glandium/git-cinnabar.git --branch master --single-branch
   export PATH=/Users/youruser/path/to/git-cinnabar:$PATH
   cd git-cinnabar && git cinnabar download

4. Alias Arcanist modified for git-cinnabar use to cinnabarc. This will allow to use both vanilla and modified Arcanist interchangeably:

   alias cinnabarc='/Users/youruser/path/to/cinnabarc/bin/arc'

5. Clone the repo(s) you plan to test on:

• hg clone https://hg.mozilla.org/automation/phabricator-qa-dev/
• git clone hg::https://hg.mozilla.org/automation/phabricator-qa-dev/ phabricator-qa-dev-cinnabar

6. In the local phabricator-qa-dev repository checkout, run:

   arc install-certificate 

Follow the instructions presented in the command line to associate your local machine with the Phabricator instance via an API key.

Repeat the process in phabricator-qa-dev-cinnabar. Use cinnabarc instead:

   cinnabarc install-certificate

7. Test away! You can create branches within your local phabricator-qa-dev repository checkout, add commits, and send them to Phabricator via arc diff .^. See the Mozilla Phabricator User Documentation for more.

This should be all you need to get going with arc and our staging and dev servers!

Test Plan

These tests will help ensure that our extensions continue to work as expected when Phabricator is upgraded. They are important as Phacility make no guarantees that internal modules/objects/APIs won't change. See the upgrade process for more.

Please update this doc with new tests as features are added/changed and as holes in the plan are discovered.

T1 - Signing up is successful

Test Plan

1. Create a new BMO account.
2. Make sure to include an irc nick in the Real Name field.
   • E.g. "John Doe [:jdoe]"
3. Go to Phabricator and login as the new user.
4. Finish account creation.

Results

1. After logging into Phabricator, on the "Create a New Account" page,
   1. Username should be prefilled with the irc nick,
   2. Real Name is correct and does not contain [] or any other extraneous characters.
2. Clicking register approval completes account creation without error.
3. The account works as expected.

T2 - Creating a revision is successful

Test Plan

1. Make sure you have arc properly setup on your machine and have run arc install-certificate using the Phabricator user you wish to test with as documented above.
2. Go to BMO and create a new bug as the Bugzilla user that the Phabricator account is connected to. (Or use an existing public bug).
   • To create bugs directly and bypass triaging, go to https://bugzilla.allizom.org/enter_bug.cgi?product=Firefox&format=__default__ (staging) or https://bugzilla-dev.allizom.org/enter_bug.cgi?product=Firefox&format=__default__ (dev)
3. Using the repo listed in the "Getting Started" section above that matches the environment you are testing, make some change to a file.
4. Run hg commit -A -m 'commit1: New changes
5. Run arc diff .^
6. Input something for the title, summary, test plan, and the correct bug id.
7. Save and exit the file.

Results

1. arc diff only submitted the 1 newly created commit.
2. Visit Phabricator and there should be a new revision with the title.
3. The revision should contain the correct diff of the changes that were made.
4. The "Bugzilla Bug ID" is correct.
   1. The Bug number is correct.
   2. The link is correct based on the environment (bugzilla.allizom.org for staging, bugzilla-dev.allizom.org for dev).
   3. If the bug in Bugzilla is a public bug, the revision should also be public.
5. Visiting the bug on Bugzilla shows an x-phabricator-request attachment.
6. Author received an email from mozphab-dev@mozilla.com. It has a title formatted as "[Differential] [Changed Policy] D{revision number}: {first line}." and a body with a text "phab-bot changed the visibility from "Administrators" to "Public (No Login Required)"."

T3 - Updating a revision with an amended commit is successful

Test Plan

Note: This would fail, please in the last step use arc diff .^ --update D{number of the revision created above} and omit the first expected result.

1. hg update to a commit which you previously submitted with arc
2. Make a change and run hg commit --amend
3. Run arc diff .^

Results

1. arc should automatically detect a revision and ask you if you want to update it.
2. The revision is updated with the new diff on Phabricator:
   1. The History tab should have two entries, Diff 1 and Diff 2.
   2. The Commits tab should have a single entry.
3. The bug id and other information remains unchanged.

T4 - Creating a secure revision is successful

Your Bugzilla user must belong to a security group, e.g. core-security.

Test Plan

1. Go to bugzilla and create a security bug:
   • Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
2. Create a new hg commit.
3. Run arc diff .^.
4. Enter the title, summary, test plan, and the bug id of the security bug.
5. Submit the revision.

Results

1. The diff and information of the revision are as expected.
2. The revision has a "Custom Policy" attached to it.
3. Click "Edit Revision" and then click on the "Visible To" drop down, and select the "Custom Policy" choice.
   • It should read "Allow members of projects", followed by the names of projects corresponding to all Bugzilla groups the private bug is categorized under. For example, a bug private to core-security, should have the project name "bmo-core-security".
4. The revision has a "secure-revision" project tag added.
5. The revision has a warning titled "This is a secure revision.".
6. The revision added the creator as a subscriber.
7. The revision is visible to the user who made it.
8. The revision is visible to users belonging to the security groups of the bug.
9. There is an x-phabricator-request attachment on the bug in Bugzilla.
10. The revision is NOT visible to the public without logging in.
11. The revision is NOT visible to logged in members without the correct permission.

T5 - Adding a secure bug to an existing revision locks it down

Your Bugzilla user must belong to a security group, e.g. core-security. You will also need a second, unprivileged user.

Test Plan

1. Create a new hg commit.
2. Run arc diff .^.
3. Enter the title, summary, and test plan. Do not include a bug number.
4. Submit the revision.
5. Check if revision is available for public.
6. Go to bugzilla and create a security bug:
   • Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
7. Edit the revision created above, and set the Bugzilla Bug ID field to the ID of the newly created bug.
8. Add the second, unprivileged Bugzilla user to the bug's CC list.

Results

1. The revision has a "Custom Policy" attached to it.
2. Click "Edit Revision" and then click on the "Visible To" drop down, and select the "Custom Policy" choice.
   • It should read "Allow members of projects", followed by the names of projects corresponding to all Bugzilla groups the private bug is categorized under. For example, a bug private to core-security, should have the project name "bmo-core-security".
3. The revision has a "secure-revision" project tag added.
4. The revision has the creator and the second Bugzilla user as subscribers.
5. The revision is visible to the user who made it.
6. The revision is visible to users belonging to the security groups of the bug.
7. The revision is NOT visible to the public without logging in.
8. The revision is NOT visible to logged in members without the correct permission.
9. The revision IS visible to the second Bugzilla user.
10. There is an x-phabricator-request attachment on the bug in Bugzilla.
11. The reviewer received an email from the author with a text "The content for this message can only be transmitted over a secure channel." and a link to Phabricator.

T6 - Creating a revision checks the bug id

Test Plan

1. Create a new hg commit.
2. Run arc diff .^
3. Enter information for the title, summary, and test plan.
4. Enter the bug id as specified for each expected result below.

Results

1. Entering an invalid bug id, e.g "abcd efg" or "$1000", fails.
2. Entering the id of a bug that does not exist fails.
3. Entering the id of a bug of a secure revision that the user does not have access to fails.
4. Entering a valid bug id is successful.

T7 - Creating multiple revisions with the same bug id is successful

Test Plan

1. Create a new hg commit.
2. Run arc diff .^
3. Enter information for the title, summary, and test plan.
4. Enter the bug id.
5. Repeat and create another revision with the same bug id.

Result

1. Both revisions are created successfully.
2. There are 2 corresponding x-phabricator-request attachments on the bug in Bugzilla.

T8 - Requesting and leaving a review on a revision is successful

Test Plan

1. Ensure that you have 2 Phabricator accounts that log in via Bugzilla ready to go.
2. Create a commit, run arc diff .^.
3. Input the title, summary, test plan, and bug id of a public bug.
4. For the "Reviewers" field enter the Phabricator username of the other account.
5. Log into Phabricator as the reviewer account.
6. Add the "Accept Revision" action at the bottom. Submit.
7. Add the "Request Changes" action at the bottom. Submit.

Results

1. The Phabricator attachment on Bugzilla is present.
2. Reviewer received an email from author with a text "{author} added a reviewer: {reviewer}"
3. Author received an email from reviewer with a text "{reviewer} accepted this revision."
4. Author received an email from reviewer with a text "{reviewer} requested changes to this revision."

T9 - Abandoning a revision obsoletes the attachment

Test Plan

1. Create a new bug.
2. Create a commit and run arc diff .^.
3. Input the title, summary, test plan, and bug id of the newly created bug.
4. Go to the newly created revision and perform the "Abandon Revision" action from the "Add Action" dropdown.

Results

1. The bug has no active attachments listed.
2. The bug has one obsolete attachment listed, linking to the created revision.

T10 - Reclaiming a revision unobsoletes the attachment

Test Plan

1. In the revision created as part of the "Abandoning a revision obsoletes the attachment" test, perform the "Reclaim revision" action.

Results

1. The bug's Phabricator attachment is unobsoleted.

T11 - Creating a Diff from local git repository to remote Mercurial repository is not allowed

Test Plan

1. Change directory to the repository phabricator-qa-dev-cinnabar.
2. Create a commit with git and run arc diff HEAD~.
3. Fill the needed data (Test Plan) and confirm.

Results

1. Creating a diff fails with :

   ERR-CONDUIT-CORE: Local VCS (git) is different from the remote repository VCS (hg).

T12 - Patching Diff created from git repository

Test Plan

1. Using a commit created above run cinnabarc diff HEAD~.
2. Fill the needed data (Test Plan) and confirm.
3. Change directory to phabricator-qa-dev.
4. Run arc patch D{number of the revision created above}.

Results

1. Code is sucessfully patched using the Diff.

T13 - Verify the private revisions deliver emails that does not contain any sensitive content

Your Bugzilla user must belong to a security group, e.g. core-security.

Test Plan

1. Login to Phabricator (after creating account in Bugzilla) using an account that can have email delivered to it, such as your own email address.
2. At the top right of Phabricator, click on your initial or gravatar image to open the drop down menu and select "Settings".
3. Click on "Email Delivery".
4. Select "Enable Self Action Mail" for the "Self Action" drop down.
5. Click "Save Changes".
6. Go to Bugzilla and create a security bug:
   • Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
7. Create a new hg commit.
8. Run arc diff .^.
9. Enter the title, summary, test plan, and the bug id of the security bug.
10. Submit the revision.

Results

1. The diff and information of the revision are as expected.
2. The revision has a "Custom Policy" attached to it.
3. The revision has a "secure-revision" project tag added.
4. The revision has a warning titled "This is a secure revision".
5. Check to see if you received an email about the new object (Revision) that was just created.
6. The email should not contain any information about the revision other than a link to Phabricator.
7. Clicking on the link in the email should take you to the Phabricator page that displays the full unfiltered email contents.
8. The email contents should contain the title, summary, test plan, reviewers, etc. of the new revision.
9. Submitting a public revision should instead show the full contents in the email similar to what was displayed on the Phabricator mail page.