QA/Photon Structure: Difference between revisions

-
No edit summary
(-)
 
(26 intermediate revisions by 2 users not shown)
Line 9: Line 9:
| Lawrence Mandel, Ryan VanderMeulen || QA Reviewer || Product Integrity || TBA || Email
| Lawrence Mandel, Ryan VanderMeulen || QA Reviewer || Product Integrity || TBA || Email
|-
|-
| Marco Mucci || EPM || Product Management || TBA || Email
| Jean Gong, Marco Mucci || EPM || Product Management || N/A || Email
|-
|-
| Gijs Kruitbosch || Software Engineer || Engineering || TBA || Email
| Jeff Griffiths, Peter Dolanjski, Justin Dolske || Engineering Manager || Product Management || 9/7/2017 || Email
|-
| Gijs Kruitbosch || Software Engineer || Engineering || 9/7/2017 || Email
|-
|-
|}
|}
Line 17: Line 19:


'''Revision History'''
'''Revision History'''
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.


{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 27: Line 31:
|-
|-
| 08/18/2017 || 1.2 || Grover Wimberly IV || Additional revisions
| 08/18/2017 || 1.2 || Grover Wimberly IV || Additional revisions
|-
| 09/01/2017 || 1.3 || Grover Wimberly IV || Includes revisions from QA managers
|-
| 09/12/2017 || 1.4 || Grover Wimberly IV || Includes revisions from Project Managers
|-
| 10/04/2017 || 1.5 || Emil Ghitta || Updates
|-
| 11/06/2017 || 1.6 || Emil Ghitta || Updates
|-
|-
|}
|}
Line 33: Line 45:
== Purpose ==
== Purpose ==


The purpose of this test plan is to  
The purpose of this test plan is to:
* Determine the scope of testing Photon Structure related bugs and features
* Determine the scope of testing Photon Structure related bugs and features
* Delegate test responsibilities between various QA personnel
* Delegate test responsibilities between various QA personnel
Line 43: Line 55:


== Scope ==
== Scope ==
The scope of this document is to test and verify new features/bugs in Photon Structure are working as intended.  
This wiki section details the testing that will be performed by the project team for the <b>Photon Structure</b> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document <u> what will be tested</u> and <u>how testing will be performed</u>.
* Testing is performed on the newest version of Nightly and tested on all three mainstream operating systems of Firefox Nightly.  
* (Windows, Mac OSX, and Linux - Ubuntu Distribution).


== Ownership ==
== Ownership ==
What resources are needed, and when (developers, QA, PMs working on the feature)
 
*Engineering lead:
** [mailto:gijskruitbosch%2Bbugs%40gmail.com Gijs Kruitbosch]
*Engineering Team:
** [mailto:mdeboer@mozilla.com Mike De Boer]
** [mailto:adw@mozilla.com Drew Willcoxon]
*UX Team:
** [mailto:abenson@mozilla.com Aaron Benson]
** [mailto:bbell@mozilla.com Bryan Bell]
 
*QA:
** Engineering Team - Nightly Testing
***[mailto:gwimberly@softvision.com Grover Wimberly IV]
** Release Team - Beta/Release Testing
*** [mailto:cornel.ionce@softvisioninc.eu Cornel Ionce] (irc:cornel_ionce) - TL for QA team
*** [mailto:emil.ghitta@softvision.ro Emil Ghitta] (irc:emilghitta) - QA
*** [mailto:ciprian.georgiu@softvision.ro Ciprian Georgiu] (irc:ciprian_georgiu) -QA peer
 
= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
Photon Structure - Including menus, buttons, toolbar, overflow panel, customization windows on all three major operating systems.
Testing will check if the functionality of the following elements work as intended:
*Menus
*Buttons
*Toolbar
*Overflow panel
*Customization window
*UI icons and elements
 
Testing will also check if the above targeted functionality elements work as expected using touch screen on all three major operating systems.


=== Out of Scope ===
=== Out of Scope ===
Animations, Preferences, Onboarding
Testing will <u>not</u> check:
*Animations
*Preferences
*Onboarding


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Windows OS, Mac OSX, and Linux (Ubuntu)
* '''Operating systems''':
 
**Windows 10
**Ubuntu 16.04
**Mac 10.12
* '''Devices''': n/a
* '''Other configuration/environmental setup needed''':
** <code>photon.structure.enabled</code> needs to be set to <code>true</code>, but it should be on by default


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
There are no special settings needed to test this feature, apart from its pref being set to true.
 
=== Nightly ===
<div class="mw-collapsible-content">
photon.structure.enabled = true (Default)
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
photon.structure.enabled = true
</div>
 
=== Post Beta / Release ===
<div class="mw-collapsible-content">
photon.structure.enabled = true
</div>
</div>


= Test Strategy =  
= Test Strategy =  
Line 102: Line 129:
| RAC-7 || Issues relating to the sidebar - switching from one side to the other. Holding it over a restart || TO-7 || 3-High || 3-Almost Certain || 3-High || 27
| RAC-7 || Issues relating to the sidebar - switching from one side to the other. Holding it over a restart || TO-7 || 3-High || 3-Almost Certain || 3-High || 27
|-
|-
| RAC-8 || Potential issues with touchscreen  || TO-8 || 2-Moderate || 2-Possible || 3-High || 12
| RAC-8 || Interface not responsive to touch gestures or input || TO-8 || 2-Moderate || 2-Possible || 3-High || 12
|}
|}


Line 117: Line 144:
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.


== Test Objectives ==
== Test Objectives ==
Line 126: Line 155:
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
|-
|-
| 1 || Bug Verification || Ensure new menus/panels open correctly || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-1 || Eng Team
| 1 || Menu Testing || Verify that the new menus/panels open correctly || Menus and Panels are functional || Manual/Performance || RAC-1 || Eng Team
|-
|-
| 2 || Bug Verification || Ensure drag and drop functionality with items from new customize window can seamlessly be swapped to location bar, relevant toolbars or overflow panel || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-2, RAC-5, RAC-6 || Eng Team
| 2 || Drag-and-drop functionality/Customize Window || Verify that the drag and drop functionality with items from new customize window can seamlessly be swapped to location bar, relevant toolbars or overflow panel || The functionality of the new customize window allows the user to customize the browser without issues || Manual/Performance || RAC-2, RAC-5, RAC-6 || Eng Team
|-
|-
| 3 || Bug Verification || Ensure the new library menu functions as intended per specification || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-3, RAC-4 || Eng Team
| 3 || Photon library menu || Verify that the new library menu is functional as per specifications || The library menu is functional and no issues occur while interacting with it|| Manual/Performance || RAC-3, RAC-4 || Eng Team
|-
|-
| 4 || Bug Verification || Ensure the new hamburger menu functions as intended per specification || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-4 || Eng Team
| 4 || Photon hamburger window || Verify that the new hamburger menu is functional as per specifications || The new hamburger menu is fully functional and it's different options can be used without issues || Manual/Performance || RAC-4 || Eng Team
|-
|-
| 5 || Bug Verification || Ensure the new page action menu functions as intended per specification || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-4 || Eng Team
| 5 || New page action window || Verify that the new page action menu is functional as per specifications || The new page action menu is functional and no issues occur || Manual/Performance || RAC-4 || Eng Team
|-
|-
| 6 || Bug Verification || Ensure the Photon Structure menus operate correctly with RTL || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-4 || Eng Team
| 6 || RTL testing || Verify that the Photon Structure menus operate correctly with RTL || The Photon Structure menus are fully functional using different RTL builds || Manual/Performance || RAC-4 || Eng Team
|-
|-
| 7 || Bug Verification || Ensure the new functionality with the sidebar works as intended || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-7 || Eng Team
| 7 || Sidebar functionality || Verify that the new sidebar functionality works as intended || The new sidebar is fully functional and no issues occur|| Manual/Performance || RAC-7 || Eng Team
|-
|-
| 8 || Bug Verification || Ensure the new structure works with touchscreen devices || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-8 || Eng Team
| 8 || Exploratory testing with touch screen || Verify the new structure works with touchscreen devices || The new structure is fully functional using touchscreen devices and no specific issues occur || Manual/Performance || RAC-8 || Eng Team
|-
|-
|}
|}


== Builds ==
== Builds ==
[https://www.mozilla.org/en-US/firefox/channel/desktop/ Up-to-date Nightly build]
This section should contain links for builds with the feature:
*[https://www.mozilla.org/en-US/firefox/channel/desktop/ Up-to-date Nightly build]
*[https://archive.mozilla.org/pub/firefox/candidates/ Beta 57 Builds]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 154: Line 185:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | June 2017 || July 2017  
|style="text-align:center;" colspan="3" | 06/2017  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | June 2017 || July 2017
|style="text-align:center;" | 06/2017 || 07/2017
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" |  June 2017 || August 2017
|style="text-align:center;" |  06/2017 || 08/2017  
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" |  August 21, 2017 || August 25, 2017  
|style="text-align:center;" |  08/21/2017 || 08/25/2017  
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | May 2017 || September 2017
|style="text-align:center;" | 05/2017 || 09/2017  
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | TBD || TBD
|style="text-align:center;" | 09/2017 || 10/2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | November 2017 with Firefox 57 || TBD
|style="text-align:center;" colspan="3" | 11/2017  
|}
|}


Line 183: Line 214:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || TBD
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1488&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Test case execution || TBD
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/1488&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || [https://bugzilla.mozilla.org/buglist.cgi?o5=notsubstring&o1=substring&o2=notsubstring&v5=qe-verify%3F&o4=notsubstring&v1=%5Bphoton-structure%5D&v2=%5Btriage%5D&v4=qe-verify-&f1=status_whiteboard&o3=notsubstring&v3=meta&f4=flagtypes.name&query_format=advanced&f3=keywords&f2=status_whiteboard&f5=flagtypes.name&list_id=13734256 Bugzilla]
| Bugs management || [https://bugzilla.mozilla.org/buglist.cgi?o5=notsubstring&o1=substring&o2=notsubstring&v5=qe-verify%3F&o4=notsubstring&v1=%5Bphoton-structure%5D&v2=%5Btriage%5D&v4=qe-verify-&f1=status_whiteboard&o3=notsubstring&v3=meta&f4=flagtypes.name&query_format=advanced&f3=keywords&f2=status_whiteboard&f5=flagtypes.name&list_id=13734256 Bugzilla]
Line 192: Line 223:
= Status =  
= Status =  
== Overview ==
== Overview ==
Track the dates and build number where feature was released to Nightly
*Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta
* Feature was merged to Beta - [https://archive.mozilla.org/pub/devedition/candidates/57.0b1-candidates/ 57.0b1 (2017-09-15)]




= References =
= References =
[https://bugzilla.mozilla.org/buglist.cgi?o5=notsubstring&o1=substring&o2=notsubstring&v5=qe-verify%3F&o4=notsubstring&v1=%5Bphoton-structure%5D&v2=%5Btriage%5D&v4=qe-verify-&f1=status_whiteboard&o3=notsubstring&v3=meta&f4=flagtypes.name&query_format=advanced&f3=keywords&f2=status_whiteboard&f5=flagtypes.name&list_id=13721695 Photon Structure Bug List]
*[https://bugzilla.mozilla.org/buglist.cgi?o5=notsubstring&o1=substring&o2=notsubstring&v5=qe-verify%3F&o4=notsubstring&v1=%5Bphoton-structure%5D&v2=%5Btriage%5D&v4=qe-verify-&f1=status_whiteboard&o3=notsubstring&v3=meta&f4=flagtypes.name&query_format=advanced&f3=keywords&f2=status_whiteboard&f5=flagtypes.name&list_id=13721695 Photon Structure Bug List]
*[https://bugzilla.mozilla.org/show_bug.cgi?id=1349210 Meta Bug - Photon - structural changes to menus/toolbars/panels]


= Testcases =  
= Testcases =  
Line 206: Line 238:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | Varies || ||  
|style="text-align:center;" | Yes || Test elements in Private Window (if applicable) ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || Default status ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" |  Depending on if tests fail || ||  
|style="text-align:center;" |  Yes || If the test does not pass with e10s enabled ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || As needed ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 221: Line 253:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | Yes || ||  
|style="text-align:center;" | Yes || As needed ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || As needed ||  
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | Varies || ||  
|style="text-align:center;" | Yes || As needed ||  
|-
|-
| Interaction (scroll, zoom)  
| Interaction (scroll, zoom)  
|style="text-align:center;" | Varies || ||  
|style="text-align:center;" | Yes || As needed (Zoom Controls in Hamburger menu and Customize windows) ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" |  Varies || ||  
|style="text-align:center;" |  Yes || As applicable ||  
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || Verifying the buttons were functioning ||  
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | Yes || ||  
|style="text-align:center;" | Yes || As needed ||  
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 245: Line 277:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easily reachable.
|style="text-align:center;" | Yes || Make sure link to support/help page exist and is easily reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | Yes || Make sure support documents are written and are correct.


|-
|-
Line 255: Line 287:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" |  Not applicable || ||  
|style="text-align:center;" |  N/A || ||  
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || As needed ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" |  Not applicable || ||  
|style="text-align:center;" |  N/A || ||  
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" |  Not applicable || ||  
|style="text-align:center;" |  N/A || ||  
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
Line 274: Line 306:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | Not applicable ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | Not applicable ||  
|style="text-align:center;" | N/A ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | Not applicable ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | Not applicable ||
|style="text-align:center;" | N/A ||


|-
|-
Line 290: Line 322:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | Not applicable ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | Not applicable ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | Not applicable ||
|style="text-align:center;" | N/A ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | Not applicable ||
|style="text-align:center;" | N/A ||


|-
|-
Line 358: Line 390:


== Test suite ==
== Test suite ==
Full Test suite - To be worked on
*Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1488&group_by=cases:section_id&group_order=asc TestRail link]


= Bug Work =
= Bug Work =
Line 400: Line 432:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


TBD
* List of OSes that will be covered by testing:<br />
** Windows 10 Surface Pro Touchscreen
** Windows 10 64bit
** Windows 7 64bit
** Ubuntu 16.04 64bit
** MacOS 10.11
* Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/6188 link]


'''Merge to Beta Sign-off'''<br />
'''Merge to Beta Sign-Off'''
 
* List of OSes that will be covered by testing<br />
TBD
** macOS 10.12
** Ubuntu 16.04 64bit
** Windows 7 64bit
** Windows 10 64bit


== Checklist ==
== Checklist ==
'''Note: August 18, 2017 -- Formatting needs fixing.'''
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
|-
|-
Line 415: Line 453:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  In-Progress  |
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   In-Progress  |
|style="text-align:center;" | {{mdone|}} 
|style="text-align:center;" |  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |  July-August 18, 2017  |
| style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |  In-Progress  |
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |  
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" | Out of Scope for Softvision |
|style="text-align:center;" | {{N/A|}}
|style="text-align:center;" |  
|style="text-align:center;"|
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | In-Progress ||  
|style="text-align:center;" | {{N/A|}}
|style="text-align:center;"|  
|-
|-
Critical/Blockers Fixed and Verified || In-Progress |  
All Defects Logged
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" |
|-
|-
|  Metrics/Telemetry||
|  Critical/Blockers Fixed and Verified
|style="text-align:center;" | Not applicable |
|style="text-align:center;" | {{in progress|}}
|style="text-align:center;" |
|style="text-align:center;" |
|-
|  Metrics/Telemetry  
|style="text-align:center;" | {{N/A|}}
|style="text-align:center;"|  
|-
|-
|  Basic/Core functionality Nightly testing
|  Basic/Core functionality Nightly testing
|style="text-align:center;" |  In-Progress |
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  QA mid-Nightly Signoff || Mid-August 2017
|  QA mid-Nightly Signoff
|style="text-align:center;" | Email to be sent |  
|style="text-align:center;"| {{mdone|}}
|style="text-align:center;" |
|style="text-align:center;" | Email Sent on 2017.08.26
|-
|-
|  QA Nightly - Full Testing  
|  QA Nightly - Full Testing  
|style="text-align:center;" | Beginning August 21, 2017 |
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" |  
|style="text-align:center;"|  
|-
|-
|  QA pre-Beta Signoff|| TBD
|  QA pre-Beta Signoff
|style="text-align:center;"|  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;"| Email sent on 2017.09.15
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | TBD |
|style="text-align:center;"| {{mdone|}}
|style="text-align:center;" |  
|-
|-
|  QA pre-Release Signoff ||  
|  QA pre-Release Signoff  
|style="text-align:center;" | TBD |
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | Email Sent on 2017.10.27
|}
|}
54

edits