View source for Phabricator/TestPlan


== Getting Started ==

'''NOTE:''' We are using the dev server for QA test runs.

The following manual steps are required to get QA up and running for Phabricator testing on Mozilla's staging server.

Alternatively, you can try out the a preset environment using Docker from this [https://github.com/dklawren/phabricator-qa Github repo]. Once checked out run:

    docker-compose build
    docker-compose run qa su - phab

You will still need to perform step 8 using your own Conduit API keys.

'''Manual Setup'''

1. Install git-cinnabar:

    git clone https://github.com/glandium/git-cinnabar.git --branch master --single-branch
    export PATH=$PATH:`pwd`/git-cinnabar
    cd git-cinnabar && git cinnabar download

2. Install the moz-phab arcanist wrapper used for submitting stacked revisions to Phabricator. You can see instructions on how to install from the [https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#setting-up-mozphab]. Usually it's enough to just <code>pip3 install MozPhab</code>.

6. Reload your .bash_profile:

    source $HOME/.bash_profile

7. Clone the repo(s) you plan to test on:

*  <code>hg clone https://hg.mozilla.org/automation/phabricator-qa-dev/</code>
*  <code>git clone hg::https://hg.mozilla.org/automation/phabricator-qa-dev/ phabricator-qa-dev-cinnabar</code>

8. In the local <code>phabricator-qa-dev</code> repository checkout, run: 

    moz-phab install-certificate 

Follow the instructions presented in the command line to associate your local machine with the Phabricator instance via an API key.

9. Test away!  You can create branches within your local <code>phabricator-qa-dev</code> repository checkout, add commits, and send them to Phabricator via <code>mozphab</code>.  See the [http://moz-conduit.readthedocs.io/en/latest/phabricator-user.html Mozilla Phabricator User Documentation] for more.  

This should be all you need to get going with moz-phab, arc, and our staging and dev servers!

== Test Plan ==

These tests will help ensure that our extensions continue to work as expected when Phabricator is upgraded.  They are important as Phacility make no guarantees that internal modules/objects/APIs won't change.  See the [[Phabricator/UpgradeProcess|upgrade process]] for more.

Please update this doc with new tests as features are added/changed and as holes in the plan are discovered.

----

=== T1 - Signing up is successful ===

'''NOTE:''' Selenium test?

====Test Plan====
# Create a new BMO account.
# Make sure to include an irc nick in the Real Name field.
#* E.g. "John Doe [:jdoe]"
# Go to Phabricator and login as the new user.
# Finish account creation.

====Results====
# After logging into Phabricator, on the "Create a New Account" page,
## Username should be prefilled with the irc nick,
## Real Name is correct and does not contain [] or any other extraneous characters.
# Clicking register approval completes account creation without error.
# The account works as expected.

----

=== T2 - Creating a revision is successful ===

====Test Plan====
# Make sure you have moz-phab properly setup on your machine and have run <code>moz-phab install-certificate</code> using the Phabricator user you wish to test with as documented above.
# Go to BMO and create a new bug as the Bugzilla user that the Phabricator account is connected to. (Or use an existing public bug).
#* To create bugs directly and bypass triaging, go to https://bugzilla.allizom.org/enter_bug.cgi?product=Firefox&format=__default__ (staging) or https://bugzilla-dev.allizom.org/enter_bug.cgi?product=Firefox&format=__default__ (dev)
# Using the repo listed in the "Getting Started" section above that matches the environment you are testing, make some change to a file.
# Run <code>hg commit -A -m 'Bug <bugid>: New changes'</code>
# Run <code>moz-phab</code>

====Results====
# <code>moz-phab</code> only submitted the 1 newly created commit.
# Visit Phabricator and there should be a new revision with the title.
# The revision should contain the correct diff of the changes that were made.
# The "Bugzilla Bug ID" is correct.
## The Bug number is correct.
## The link is correct based on the environment (bugzilla.allizom.org for staging, bugzilla-dev.allizom.org for dev).
## If the bug in Bugzilla is a public bug, the revision should also be public.
# Visiting the bug on Bugzilla shows an entry for the new revision in the Phabricator Revisions section.
# Author received an email from <code>mozphab-dev@mozilla.com</code>. It has a title formatted as "[Differential] [Changed Policy] D{revision number}: {first line}." and a body with a text "phab-bot changed the visibility from "Administrators" to "Public (No Login Required)"."

----

=== T3 - Updating a revision with an amended commit is successful ===

====Test Plan====

# <code>hg update</code> to a commit which you previously submitted with moz-phab
# Make a change and run <code>hg commit --amend</code>
# Run <code>moz-phab</code>

====Results====
# moz-phab should automatically detect a current revision and update it.
# The revision is updated with the new diff on Phabricator:
## The History tab should have two entries, Diff 1 and Diff 2.
## The Commits tab should have a single entry.
# The bug id and other information remains unchanged.

----

=== T4 - Creating a secure revision is successful ===

Your Bugzilla user must belong to a security group, e.g. core-security.

====Test Plan====
# Go to bugzilla and create a security bug:
# Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
# Run <code>hg commit -A -m 'Bug <bugid>: Private changes'</code>
# Run <code>moz-phab</code>.

====Results====
# The diff and information of the revision are as expected.
# The revision has a "Custom Policy" attached to it.
# Click "Edit Revision" and then click on the "Visible To" drop down, and select the "Custom Policy" choice.
# It should read "Allow members of projects", followed by the names of projects corresponding to all Bugzilla groups the private bug is categorized under. For example, a bug private to core-security, should have the project name "bmo-core-security".
# The revision has a "secure-revision" project tag added.
# The revision has a warning titled "This is a secure revision.".
# The revision added the creator as a subscriber.
# The revision is visible to the user who made it.
# The revision is visible to users belonging to the security groups of the bug.
# Visiting the bug on Bugzilla shows an entry for the new revision in the Phabricator Revisions section with summary as '(secure)'.
# The revision is NOT visible to the public without logging in.
# The revision is NOT visible to logged in members without the correct permission.

----

=== T5 - Adding a secure bug to an existing revision locks it down ===

Your Bugzilla user must belong to a security group, e.g. core-security.  You will also need a second, unprivileged user.

====Test Plan====
# Go to bugzilla and create a public bug:
# Run <code>hg commit -A -m 'Bug <bugid>: Public changes'</code>
# Run <code>moz-phab</code>.
# Check if revision is available for public.
# Go to bugzilla and create a security bug:
# Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
# Edit the revision created above, and set the Bugzilla Bug ID field to the ID of the newly created private bug.
# Add the second, unprivileged Bugzilla user to the bug's CC list.

====Results====
# The revision has a "Custom Policy" attached to it.
# Click "Edit Revision" and then click on the "Visible To" drop down, and select the "Custom Policy" choice.
# It should read "Allow members of projects", followed by the names of projects corresponding to all Bugzilla groups the private bug is categorized under. For example, a bug private to core-security, should have the project name "bmo-core-security".
# The revision has a "secure-revision" project tag added.
# The revision has the creator and the second Bugzilla user as subscribers.
# The revision is visible to the user who made it.
# The revision is visible to users belonging to the security groups of the bug.
# The revision is NOT visible to the public without logging in.
# The revision is NOT visible to logged in members without the correct permission.
# The revision IS visible to the second Bugzilla user.
# Visiting the bug on Bugzilla shows an entry for the new revision in the Phabricator Revisions section.
# The reviewer received an email from the author with a text "The content for this message can only be transmitted over a secure channel." and a link to Phabricator.

----

=== T6 - Creating a revision checks the bug id ===

====Test Plan====
# Enter the bug id as specified for each expected result below.
# Run <code>hg commit -A -m 'Bug <bugid>: Public changes'</code>
# Run <code>moz-phab</code>.

====Results====
# Entering an invalid bug id, e.g "abcd efg" or "$1000", fails.
# Entering the id of a bug that does not exist fails.
# Entering the id of a bug of a secure revision that the user does not have access to fails.
# Entering a valid bug id is successful.

----

=== T7 - Creating multiple revisions with the same bug id is successful ===

====Test Plan====
# Run <code>hg commit -A -m 'Bug <bugid>: Public changes'</code>
# Run <code>moz-phab</code>.
# Create a different comment using same bug id but different title.

====Result====
# Both revisions are created successfully.
# Visiting the bug on Bugzilla shows 2 corresponding entries for the revisions in the Phabricator Revisions section.

----

=== T8 - Requesting and leaving a review on a revision is successful ===

====Test Plan====
# Ensure that you have 2 Phabricator accounts that log in via Bugzilla ready to go.
# Create a commit with other account as reviewer using <code>hg commit -m 'Bug <bugid>: New changes r?<reviewer>'</code>.
# Run <code>moz-phab</code>.
# Log into Phabricator as the reviewer account.
# Add the "Accept Revision" action at the bottom. Submit.
# Add the "Request Changes" action at the bottom. Submit.

====Results====
# The Phabricator revision is visible in the Bugzilla bug.
# Reviewer received an email from author with a text "{author} added a reviewer: {reviewer}"
# Author received an email from reviewer with a text "{reviewer} accepted this revision."
# Author received an email from reviewer with a text "{reviewer} requested changes to this revision."

----

=== T9 - Abandoning a revision obsoletes the attachment ===

'''NOTE:''' Selenium test?

====Test Plan====
# On a previously created revision, perform the "Abandon Revision" action from the "Add Action" dropdown.

====Results====
# The bug shows the revision as 'Abandoned' in the Phabricator Revisions list of the bug.
# The bug history shows the attachment is being set to obsolete.

----

=== T10 - Reclaiming a revision unobsoletes the attachment ===

'''NOTE:''' Selenium test?

====Test Plan====
# In the revision used as part of the "Abandoning a revision obsoletes the attachment" test, perform the "Reclaim revision" action.

====Results====
# The bug's Phabricator attachment is unobsoleted and the revision is not abandoned any longer in the Phabricator revisions list.

----

=== T12 - Patching Diff created from git repository ===

====Test Plan====
# Using a commit created above run <code>moz-phab HEAD~</code>.
# Fill the ccommit message and confirm.
# Change directory to <code>phabricator-qa-dev</code>.
# Run <code>moz-phab patch D{number of the revision created above}</code>.

====Results====
# Code is sucessfully patched using the Diff.

----

=== T13 - Verify the private revisions deliver emails that does not contain any sensitive content ===

'''NOTE:''' Selenium test?

Your Bugzilla user must belong to a security group, e.g. core-security.

====Test Plan====
# Login to Phabricator (after creating account in Bugzilla) using an account that can have email delivered to it, such as your own email address.
# At the top right of Phabricator, click on your initial or gravatar image to open the drop down menu and select "Settings".
# Click on "Email Delivery".
# Select "Enable Self Action Mail" for the "Self Action" drop down.
# Click "Save Changes".
# Go to Bugzilla and create a security bug:
#* Click "Edit Bug", open the "Security" panel, and check one of the security-sensitive boxes, e.g. "Security-Sensitive Core Bug".
# Create a new hg commit.
# Run <code>moz-phab</code>.

====Results====
# The diff and information of the revision are as expected.
# The revision has a "Custom Policy" attached to it.
# The revision has a "secure-revision" project tag added.
# The revision has a warning titled "This is a secure revision".
# Check to see if you received an email about the new object (Revision) that was just created.
# The email should not contain any information about the revision other than a link to Phabricator.
# Clicking on the link in the email should take you to the Phabricator page that displays the full unfiltered email contents.
# The email contents should contain the title, summary, test plan, reviewers, etc. of the new revision.
# Submitting a public revision should instead show the full contents in the email similar to what was displayed on the Phabricator mail page.

=== T14 - Verify that a reverted commit will reopen the associated revision ===

====Test Plan====
# Clone the following repositories to the same directory on your machine: <code>hg clone https://hg.mozilla.org/conduit-testing/vct vct</code>
# Configure your <code>.ssh/config</code> to contain the following: <code>Host hg.mozilla.org\n  User dlawrence@mozilla.com </code>
# Copy your SSH keys to the .ssh directory and set the appropriate permissions.
# Change into the new repository <code>vct</code>.
# Update `.hg/hgrc` to add the following line to the `[paths]` section: `default:pushurl = ssh://hg.mozilla.org/conduit-testing/vct`
# Make sure you have moz-phab properly setup on your machine and have run <code>moz-phab install-certificate</code> using the Phabricator user you wish to test with as documented above.
# Go to BMO and create a new bug as the Bugzilla user that the Phabricator account is connected to. (Or use an existing public bug). 
# Make some change to a file.
# Run <code>hg commit -A -m 'Bug <bugid>: New changes'</code>
# Run <code>moz-phab</code> to create a new revision.
# As the review user, accept the changes in the new revision.
# As the patch author run <code>hg push</code> to push the changes and auto-close the revision.
# To backout the most recent commit, run <code>hg backout .</code> and accept the default commit message.
# Push the new backed out commit by running <code>hg push</code>.

====Results====
# The diff and information of the revision are as expected.
# When <code>hg push</code> is executed, after a period of time, the revision should automatically transition to closed.
# When <code>hg backout</code> commit is pushed, after a period of time, the revision should automatically reopen.

=== T15 - Verify that the Phabricator Email API endpoint is producing correct output ===

To verify that the <code>feed.for_email.query</code> API endpoint is still stable, we can invoke it to get the email events for
a full patch workflow. If the resulting payload hasn't changed, then we know that there hasn't been a regression.
The two patches that we are using are:

* https://phabricator-dev.allizom.org/D2942
* https://phabricator-dev.allizom.org/D2943 (secure bug)

======Note======
The events we are expecting are going to be slightly different than the "real" events that would be generated in
real-time. This is because the state of the revision (such as current reviewers, current bug, etc) are resolved
based on its status at API-call time - '''not''' its status when the event was triggered.

To avoid this inaccuracy, testing the API endpoint would require manually performing all the actions and calling
the endpoint immediately after each one, which would be very tedious.

====Test Plan====
# If you don't already have one, get an API Token for email-bot: https://phabricator-dev.allizom.org/settings/user/email-bot/page/apitokens/.
# If not already available, install [https://stedolan.github.io/jq/ jq].
# Clone the Arcanist code to your current directory: `git clone https://github.com/mozilla-conduit/arcanist.git arcanist`
# From the command-line, invoke:

<syntaxhighlight lang="bash">
$ echo '{
  "storyLimit": 39,
  "after": 6853094612582369220
}' | arcanist/bin/arc call-conduit --conduit-uri https://phabricator-dev.allizom.org/ --conduit-token <your-token> feed.for_email.query -- \
   | jq -r '.response' | jq . > output.txt
</syntaxhighlight>

====Results====
# Copy the [[Phabricator/TestPlan/ExpectationT15|expected payload]] and put it into a file, such as <code>expected.txt</code>
# Check that the file files are identical:

<syntaxhighlight lang="bash">
$ diff <(jq -S . expected.txt) <(jq -S . output.txt)
</syntaxhighlight>

or on Windows, if "jq" is installed:

<syntaxhighlight>
$ jq -S . output.txt > output-sorted.txt
$ jq -S . expected.txt > expected-sorted.txt
$ fc output-sorted.txt expected-sorted.txt
</syntaxhighlight>