MozillaWiki

QA/Photon Structure: Difference between revisions

Page Discussion

QA/Photon Structure (view source)

Revision as of 17:55, 6 November 2017

3,188 bytes added ,  6 November 2017
-
(-)
 
(17 intermediate revisions by 2 users not shown)
Line 9: Line 9:
| Lawrence Mandel, Ryan VanderMeulen || QA Reviewer || Product Integrity || TBA || Email
| Lawrence Mandel, Ryan VanderMeulen || QA Reviewer || Product Integrity || TBA || Email
|-
|-
| Marco Mucci || EPM || Product Management || TBA || Email
| Jean Gong, Marco Mucci || EPM || Product Management || N/A || Email
|-
|-
| Gijs Kruitbosch || Software Engineer || Engineering || TBA || Email
| Jeff Griffiths, Peter Dolanjski, Justin Dolske || Engineering Manager || Product Management || 9/7/2017 || Email
|-
| Gijs Kruitbosch || Software Engineer || Engineering || 9/7/2017 || Email
|-
|-
|}
|}
Line 17: Line 19:


'''Revision History'''
'''Revision History'''
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.


{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:60%"
Line 27: Line 31:
|-
|-
| 08/18/2017 || 1.2 || Grover Wimberly IV || Additional revisions
| 08/18/2017 || 1.2 || Grover Wimberly IV || Additional revisions
|-
| 09/01/2017 || 1.3 || Grover Wimberly IV || Includes revisions from QA managers
|-
| 09/12/2017 || 1.4 || Grover Wimberly IV || Includes revisions from Project Managers
|-
| 10/04/2017 || 1.5 || Emil Ghitta || Updates
|-
| 11/06/2017 || 1.6 || Emil Ghitta || Updates
|-
|-
|}
|}
Line 33: Line 45:
== Purpose ==
== Purpose ==


The purpose of this test plan is to  
The purpose of this test plan is to:
* Determine the scope of testing Photon Structure related bugs and features
* Determine the scope of testing Photon Structure related bugs and features
* Delegate test responsibilities between various QA personnel
* Delegate test responsibilities between various QA personnel
Line 43: Line 55:


== Scope ==
== Scope ==
The scope of this document is to test and verify new features/bugs in Photon Structure are working as intended.  
This wiki section details the testing that will be performed by the project team for the <b>Photon Structure</b> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document <u> what will be tested</u> and <u>how testing will be performed</u>.
* Testing is performed on the newest version of Nightly and tested on all three mainstream operating systems of Firefox Nightly.  
* (Windows, Mac OSX, and Linux - Ubuntu Distribution).


== Ownership ==
== Ownership ==


Engineering lead:
*Engineering lead:
  [mailto:gijskruitbosch%2Bbugs%40gmail.com Gijs Kruitbosch]
** [mailto:gijskruitbosch%2Bbugs%40gmail.com Gijs Kruitbosch]
 
*Engineering Team:
Engineering Team:
** [mailto:mdeboer@mozilla.com Mike De Boer]
  [mailto:mdeboer@mozilla.com Mike De Boer]
** [mailto:adw@mozilla.com Drew Willcoxon]
  [mailto:adw@mozilla.com Drew Willcoxon]
*UX Team:
** [mailto:abenson@mozilla.com Aaron Benson]
** [mailto:bbell@mozilla.com Bryan Bell]


UX Team:
*QA:
  [mailto:abenson@mozilla.com Aaron Benson]
** Engineering Team - Nightly Testing
  [mailto:bbell@mozilla.com Bryan Bell]
***[mailto:gwimberly@softvision.com Grover Wimberly IV]
 
** Release Team - Beta/Release Testing
QA:
*** [mailto:cornel.ionce@softvisioninc.eu Cornel Ionce] (irc:cornel_ionce) - TL for QA team
  [mailto:gwimberly@softvision.com Grover Wimberly IV]
*** [mailto:emil.ghitta@softvision.ro Emil Ghitta] (irc:emilghitta) - QA
*** [mailto:ciprian.georgiu@softvision.ro Ciprian Georgiu] (irc:ciprian_georgiu) -QA peer


= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
Photon Structure - Including menus, buttons, toolbar, overflow panel, customization windows on all three major operating systems.
Testing will check if the functionality of the following elements work as intended:
*Menus
*Buttons
*Toolbar
*Overflow panel
*Customization window
*UI icons and elements
 
Testing will also check if the above targeted functionality elements work as expected using touch screen on all three major operating systems.


=== Out of Scope ===
=== Out of Scope ===
Animations, Preferences, Onboarding, Mobile Environments
Testing will <u>not</u> check:
*Animations
*Preferences
*Onboarding


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Windows 10
* '''Operating systems''':
Ubuntu 16.04
**Windows 10
Mac 10.12
**Ubuntu 16.04
**Mac 10.12
* '''Devices''': n/a
* '''Other configuration/environmental setup needed''':
** <code>photon.structure.enabled</code> needs to be set to <code>true</code>, but it should be on by default


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
There are no special settings needed to test this feature, apart from its pref being set to true.
 
=== Nightly ===
<div class="mw-collapsible-content">
photon.structure.enabled = true (Default)
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
photon.structure.enabled = true
</div>
 
=== Post Beta / Release ===
<div class="mw-collapsible-content">
photon.structure.enabled = true
</div>
</div>


= Test Strategy =  
= Test Strategy =  
Line 117: Line 129:
| RAC-7 || Issues relating to the sidebar - switching from one side to the other. Holding it over a restart || TO-7 || 3-High || 3-Almost Certain || 3-High || 27
| RAC-7 || Issues relating to the sidebar - switching from one side to the other. Holding it over a restart || TO-7 || 3-High || 3-Almost Certain || 3-High || 27
|-
|-
| RAC-8 || Potential issues with touchscreen  || TO-8 || 2-Moderate || 2-Possible || 3-High || 12
| RAC-8 || Interface not responsive to touch gestures or input || TO-8 || 2-Moderate || 2-Possible || 3-High || 12
|}
|}


Line 132: Line 144:
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
'''RAC:''' Refers to the "Risk Assessment and Coverage" item that is covering.


== Test Objectives ==
== Test Objectives ==
Line 141: Line 155:
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
|-
|-
| 1 || Bug Verification || Ensure new menus/panels open correctly || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-1 || Eng Team
| 1 || Menu Testing || Verify that the new menus/panels open correctly || Menus and Panels are functional || Manual/Performance || RAC-1 || Eng Team
|-
|-
| 2 || Bug Verification || Ensure drag and drop functionality with items from new customize window can seamlessly be swapped to location bar, relevant toolbars or overflow panel || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-2, RAC-5, RAC-6 || Eng Team
| 2 || Drag-and-drop functionality/Customize Window || Verify that the drag and drop functionality with items from new customize window can seamlessly be swapped to location bar, relevant toolbars or overflow panel || The functionality of the new customize window allows the user to customize the browser without issues || Manual/Performance || RAC-2, RAC-5, RAC-6 || Eng Team
|-
|-
| 3 || Bug Verification || Ensure the new library menu functions as intended per specification || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-3, RAC-4 || Eng Team
| 3 || Photon library menu || Verify that the new library menu is functional as per specifications || The library menu is functional and no issues occur while interacting with it|| Manual/Performance || RAC-3, RAC-4 || Eng Team
|-
|-
| 4 || Bug Verification || Ensure the new hamburger menu functions as intended per specification || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-4 || Eng Team
| 4 || Photon hamburger window || Verify that the new hamburger menu is functional as per specifications || The new hamburger menu is fully functional and it's different options can be used without issues || Manual/Performance || RAC-4 || Eng Team
|-
|-
| 5 || Bug Verification || Ensure the new page action menu functions as intended per specification || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-4 || Eng Team
| 5 || New page action window || Verify that the new page action menu is functional as per specifications || The new page action menu is functional and no issues occur || Manual/Performance || RAC-4 || Eng Team
|-
|-
| 6 || Bug Verification || Ensure the Photon Structure menus operate correctly with RTL || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-4 || Eng Team
| 6 || RTL testing || Verify that the Photon Structure menus operate correctly with RTL || The Photon Structure menus are fully functional using different RTL builds || Manual/Performance || RAC-4 || Eng Team
|-
|-
| 7 || Bug Verification || Ensure the new functionality with the sidebar works as intended || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-7 || Eng Team
| 7 || Sidebar functionality || Verify that the new sidebar functionality works as intended || The new sidebar is fully functional and no issues occur|| Manual/Performance || RAC-7 || Eng Team
|-
|-
| 8 || Bug Verification || Ensure the new structure works with touchscreen devices || Ensuring the bug is fixed on major desktop operating systems || Manual/Performance || RAC-8 || Eng Team
| 8 || Exploratory testing with touch screen || Verify the new structure works with touchscreen devices || The new structure is fully functional using touchscreen devices and no specific issues occur || Manual/Performance || RAC-8 || Eng Team
|-
|-
|}
|}


== Builds ==
== Builds ==
[https://www.mozilla.org/en-US/firefox/channel/desktop/ Up-to-date Nightly build]
This section should contain links for builds with the feature:
*[https://www.mozilla.org/en-US/firefox/channel/desktop/ Up-to-date Nightly build]
*[https://archive.mozilla.org/pub/firefox/candidates/ Beta 57 Builds]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 169: Line 185:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 06/2017 || 07/2017  
|style="text-align:center;" colspan="3" | 06/2017  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
Line 184: Line 200:
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | TBD || TBD
|style="text-align:center;" | 09/2017 || 10/2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | 11/2017 with Firefox 57 || TBD
|style="text-align:center;" colspan="3" | 11/2017  
|}
|}


Line 198: Line 214:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || TBD
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1488&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Test case execution || TBD
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/suites/view/1488&group_by=cases:section_id&group_order=asc TestRail]
|-
|-
| Bugs management || [https://bugzilla.mozilla.org/buglist.cgi?o5=notsubstring&o1=substring&o2=notsubstring&v5=qe-verify%3F&o4=notsubstring&v1=%5Bphoton-structure%5D&v2=%5Btriage%5D&v4=qe-verify-&f1=status_whiteboard&o3=notsubstring&v3=meta&f4=flagtypes.name&query_format=advanced&f3=keywords&f2=status_whiteboard&f5=flagtypes.name&list_id=13734256 Bugzilla]
| Bugs management || [https://bugzilla.mozilla.org/buglist.cgi?o5=notsubstring&o1=substring&o2=notsubstring&v5=qe-verify%3F&o4=notsubstring&v1=%5Bphoton-structure%5D&v2=%5Btriage%5D&v4=qe-verify-&f1=status_whiteboard&o3=notsubstring&v3=meta&f4=flagtypes.name&query_format=advanced&f3=keywords&f2=status_whiteboard&f5=flagtypes.name&list_id=13734256 Bugzilla]
Line 207: Line 223:
= Status =  
= Status =  
== Overview ==
== Overview ==
Track the dates and build number where feature was released to Nightly
*Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta
* Feature was merged to Beta - [https://archive.mozilla.org/pub/devedition/candidates/57.0b1-candidates/ 57.0b1 (2017-09-15)]




= References =
= References =
[https://bugzilla.mozilla.org/buglist.cgi?o5=notsubstring&o1=substring&o2=notsubstring&v5=qe-verify%3F&o4=notsubstring&v1=%5Bphoton-structure%5D&v2=%5Btriage%5D&v4=qe-verify-&f1=status_whiteboard&o3=notsubstring&v3=meta&f4=flagtypes.name&query_format=advanced&f3=keywords&f2=status_whiteboard&f5=flagtypes.name&list_id=13721695 Photon Structure Bug List]
*[https://bugzilla.mozilla.org/buglist.cgi?o5=notsubstring&o1=substring&o2=notsubstring&v5=qe-verify%3F&o4=notsubstring&v1=%5Bphoton-structure%5D&v2=%5Btriage%5D&v4=qe-verify-&f1=status_whiteboard&o3=notsubstring&v3=meta&f4=flagtypes.name&query_format=advanced&f3=keywords&f2=status_whiteboard&f5=flagtypes.name&list_id=13721695 Photon Structure Bug List]
*[https://bugzilla.mozilla.org/show_bug.cgi?id=1349210 Meta Bug - Photon - structural changes to menus/toolbars/panels]


= Testcases =  
= Testcases =  
Line 221: Line 238:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | Yes || ||  
|style="text-align:center;" | Yes || Test elements in Private Window (if applicable) ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || Default status ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || If the test does not pass with e10s enabled ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || As needed ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 236: Line 253:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | Yes || ||  
|style="text-align:center;" | Yes || As needed ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || As needed ||  
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | Yes || ||  
|style="text-align:center;" | Yes || As needed ||  
|-
|-
| Interaction (scroll, zoom)  
| Interaction (scroll, zoom)  
|style="text-align:center;" | Yes || ||  
|style="text-align:center;" | Yes || As needed (Zoom Controls in Hamburger menu and Customize windows) ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || As applicable ||  
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || Verifying the buttons were functioning ||  
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | Yes || ||  
|style="text-align:center;" | Yes || As needed ||  
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 270: Line 287:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" |  N/A || ||  
|style="text-align:center;" |  N/A || ||  
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" |  Yes || ||  
|style="text-align:center;" |  Yes || As needed ||  
|-
|-
| Requires install testing     
| Requires install testing     
Line 305: Line 322:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | No || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
Line 373: Line 390:


== Test suite ==
== Test suite ==
Full Test suite - To be worked on
*Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1488&group_by=cases:section_id&group_order=asc TestRail link]


= Bug Work =
= Bug Work =
Line 415: Line 432:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


TBD
* List of OSes that will be covered by testing:<br />
 
** Windows 10 Surface Pro Touchscreen
'''Merge to Beta Sign-off'''<br />
** Windows 10 64bit
** Windows 7 64bit
** Ubuntu 16.04 64bit
** MacOS 10.11
* Link for the tests run -[https://testrail.stage.mozaws.net/index.php?/plans/view/6188 link]


TBD
'''Merge to Beta Sign-Off'''
* List of OSes that will be covered by testing<br />
** macOS 10.12
** Ubuntu 16.04 64bit
** Windows 7 64bit
** Windows 10 64bit


== Checklist ==
== Checklist ==
Line 431: Line 457:
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" | {{mdone|}}  ||  
|style="text-align:center;" | {{mdone|}}   
|style="text-align:center;" |  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |  {{in progress|}} ||  
| style="text-align:center;" |  {{mdone|}}  
|style="text-align:center;" |  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |  TBA ||  
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |  
|-
|-
|  Automation Coverage |
|  Automation Coverage  
|style="text-align:center;" | {{N/A|}} ||
|style="text-align:center;" | {{N/A|}}  
|style="text-align:center;"|
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | {{N/A|}} ||  
|style="text-align:center;" | {{N/A|}}  
|style="text-align:center;"|  
|-
|-
|  All Defects Logged  
|  All Defects Logged  
|style="text-align:center;" | {{in progress|}}  
|style="text-align:center;" | {{mdone|}}  
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
Line 453: Line 484:
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry  
|style="text-align:center;" |  
|style="text-align:center;" | {{N/A|}}
|style="text-align:center;"|  
|-
|-
|  Basic/Core functionality Nightly testing
|  Basic/Core functionality Nightly testing
|style="text-align:center;" |  {{in progress|}}  
|style="text-align:center;" |  {{mdone|}}  
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  QA mid-Nightly Signoff||  
|  QA mid-Nightly Signoff
|style="text-align:center;" | Email to be sent
|style="text-align:center;"| {{mdone|}}
|style="text-align:center;" | Email Sent on 2017.08.26
|-
|-
|  QA Nightly - Full Testing  
|  QA Nightly - Full Testing  
|style="text-align:center;" | {{in progress|}} ||  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;"|  
|-
|-
|  QA pre-Beta Signoff||  
|  QA pre-Beta Signoff
|style="text-align:center;"| Email to be sent  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;"| Email sent on 2017.09.15
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | Email to be sent
|style="text-align:center;"| {{mdone|}}
|style="text-align:center;" |
|-
|-
|  QA pre-Release Signoff ||  
|  QA pre-Release Signoff  
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | Email Sent on 2017.10.27
|}
|}
Eghitta
54

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1178847...1183545"