QA/Mobile/FocusAndroidTestPlan

This is the Test Plan for Focus for Android (https://github.com/mozilla-mobile/focus-android)

Intro/Summary/Notes

Intro and Summary

• This wiki outlines the test requirements for Focus for Android releases.
• Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases

Notes

• FAQs

Feature Lists

For the list of features and its status, please refer to here. 'Aqua' label refers to the issues that are slated for the next upcoming release

• NOTE: Focus for Android bugs are maintained in Github

Schedule

• All features for 2.0 should be completed by Aug 4th, 2017 to begin acceptance testing
• Testing, including acceptance testing, should be completed by Aug 10th, 2017.
• Focus for Android 2.0 is currently scheduled to be released Aug 17th, 2017.

Personnel

Program Management

• Jenn Chaulk

Product Management

• Barbara Bermes

Development

• Sebastian Kaspari
• Michael Comella
• Chenxia Liu

QA Team

• No-Jun Park

Resource Plan

• Each QA person will be 40% allocated to this project

Test Strategy

Process

• When the development work on github issue is completed, whether it is before/after code freeze, ReadyForQA label will be applied so that the issue is available for testing. The issue should belong to a correct milestone before the development work can be started.
• QA will verify the issue, and upon completion, it'll be marked with VerifiedByQA, removing ReadyForQA label.
• If an issue is not QA verifiable, it can be marked with QANotNeeded
• After code freeze, ideally, all issues should be marked with either ReadyForQA, VerifiedByQA, or QANotNeeded.
• After the QA signoff, ideally, all issues should be marked with VerifiedByQA, or QANotNeeded.
• The issue can be closed after it is marked as VerifiedByQA, or QANotNeeded
• This does not apply to metabugs.

Manual Tests

UI Test suite for Focus for Android will be located in TestRail. As a minimum, each of the P1 UI features will have corresponding test cases. The test suite will be executed by Softvision for the acceptance testing.

Automated Tests

Automation tests will be mainly used for quick regression check of key UI features. Overall, automation tests will be added to the same Github repo, and executed on BuddyBuild CI system. While the unit tests will be run for each commit, UI Tests (in Espresso/UIAutomator framework) will be executed in master branch only. The test result will be inspected for new failures, and github issue will be raised for tracking.

Automation tests are located here.

In addition to regularly run automated tests, QA will also generate screenshots for l10n verification.

Code Analysis

• Codecov is added to the github repo, and will display increase/decrease of unit test code coverage
• Findbugs addon is added to warn developers against possible code issues, as well as Lint.

Outside Scope

Following tests are outside the scope of the QA:

• Unit Tests
• Performance Tests
• Localization Tests
• Beta testing with a wider audience

Test Deliverables

• Completed TestRail Test suite which covers all features with UI aspects
• Automated Test suite in master branch, running on every checkin
• Github issues for every issue found
• Manual acceptance test report
• Signoff Decision

Testing Days

June 14th 2017:

• Ran Functional test suites on both Android phone (Samsung Galaxy S6 EDGE - Android 7.0) and tablet (Nexus 9 - Android 6.0.1)
  • Test rail suite for Nexus 9 Android 6.0.1
    
  • Test rail suite for Samsung Galaxy S6 EDGE Android 7.0
• 2 New issues added to the mozilla-mobile/focus-android github section
  • Screenshot block notification not present on tablet devices - this one is a little bit confusing for the user
    
  • Focus Crashes if you erase history from android bar while site loading - this one, IMO should be resolved ASAP since it crashes the App pretty easily
    

July 14th, 2017 (Ioana Chiorean):

• Ran Functional test suite on:
  • Nexus 7 - Android 5.1.2
  • Pixel - Android 7.1.2
  • Nexus 6P - Android 8.0
• New issues added to the mozilla-mobile/focus-android github section
  • Thumbnail in Recent Apps shows settings menu #924
  • White screen while playing Vimeo videos #925
  • Vimeo.com displayed blank on Nexus 7 ( 5.1.2) #926
• Issues investigated/commented the mozilla-mobile/focus-android github section
  • Filter out Focus in the BottomSheet when "Open with..." is clicked #894

July 14th, 2017 (Oana Horvath):

• Ran Functional test suite on: Huawei Honor 8 (Android 6.0)
• New issues added to the mozilla-mobile/focus-android github section
  • Focus 1.1. Beta Build #2202 - always prompts that there is an update available, opens #2196
  • Invalid icons of share buttons on Nytimes video section

August 24th/25th, 2017 (Ioana Chiorean):

• Ran Functional test suite on Pixel with Android 8.0
• New issues:
  • Unable to open comments page on reddit ( with some workaround)
  • Tapping URL bar after search should highlight the searched terms
• Chrome webstore
  • When you have tracking off on https://blockads.fivefilters.org/ you get a suggestion from chrome webstore - Is our UA not recognized?
• TC Not Clear
  • Search on URL bar
    • Step 2 - URL bar displays the search URL
  • Open a link in Focus from another app
    • Step 2 - If focus was open before with something else, that page is displayed
  • Enable/Disable Trackers
    • Step 3 - If the trackers i soff, there is a - display and not 0
  • Enable/Disable Stealth Mode
    • Step 2 - If the stealth mode is disabled by default and we do not ask the user to enable it you are able to take the screenshot.
• Still reproducing:
  • Vimeo.com displayed blank on Nexus 7 ( 5.1.2) #926
  • Focus 1.1. Beta Build #2202 - always prompts that there is an update available, opens #2196

September 9th, 2017 (Ioana Chiorean):

• Ran Functional test suite on Pixel with Android 8.0
• New logged issues:
  • Open with Firefox (Nightly etc) not present for strava.com #1298
  • about:rights not localized #1299
  • Mozilla menu not localized #1300
• TC Not Clear
  • Enable/Disable Stealth Mode
    • Step 2 - If the stealth mode is disabled by default and we do not ask the user to enable it you are able to take the screenshot.
  • Open New Tab from Other Apps
    • Step 3 - if we close and restart the Focus app - old tabs are not displayed.
  • Erase from Notification Drop-Down
    • tap 'Erase browsing history' is now "Erase and Open" so the browser will be open & present in recent app list
• Message changed:
  • "Your Browsing History has been Deleted" - "Your browsing history has been erased"
• Commented:
  • Focus 1.1. Beta Build #2202 - always prompts that there is an update available, opens #2196

December 6th, 2017 (Sorina Florean):

• Ran Functional test suite on Nexus 5 with Android 6.0.1
• New logged issues:
  • Video exit full screen mode when device orientation is changed #1926
  • Trash can icon active when no search engine are selected #1917
  • Inconsistency when closing pages from Your Rights #1920
  • Settings Menu not displayed properly #1922
• Issues still reproducible:
  • Lastpass is not filtering the autofill suggestion / only shows first few sites #1883

Dependencies/Risks

• Since the blocking status of trackers is not exposed via adb log or UI, its validation will rely on unit tests
• The performance of Focus for Android is not measured precisely, will rely on the tester feedback
• The tester pool for a preliminary first release will be extremely small
• The automation/manual test suites will be built incrementally as new features are added to Github, and the late features may not be exposed to validation as much as earlier ones. The acceptance test after the feature completion date will be performed with the special emphasis on 1. key features and 2. most recently added features

Signoff/Exit Criteria

• No critical or major UI bugs
• No easily reproducible crashes