MozillaWiki

QA/Photon Onboarding Tour Overlay: Difference between revisions

Page Discussion

QA/Photon Onboarding Tour Overlay (view source)

Revision as of 18:30, 13 September 2017

1,818 bytes added ,  13 September 2017
→‎Environments
 
(37 intermediate revisions by the same user not shown)
Line 16: Line 16:
= Overview =
= Overview =
== Purpose ==
== Purpose ==
Gives a new user/user that has not used Firefox in a while a description and an introduction to Firefox's new innovative features, such as:
Gives a new user/user that has not used Firefox in a while a description and an introduction to Firefox's new innovative features, such as: <br />
 
'''V1 New User'''
* Private Browsing
* Private Browsing
* Addons
* Addons
Line 24: Line 26:
* Sync
* Sync


with a overlay that is accessed by clicking the fox head on the about:newtab page or the default about:home page. The overlay is also accessed by clicking the Onboarding Tour Notification.
'''V2 New User'''
* Performance
* Private Browsing
* Screenshots
* Add-ons
* Customize
* Default Browser
 
'''V2 Updated User'''
* Performance
* Library
* Screenshots
* Search
* Customize
* Sync


== Scope ==
with a overlay that is accessed by clicking the Firefox icon or the watermark on the about:newtab page or the default about:home page. The overlay is also accessed by clicking the Onboarding Tour Notification.
Photon Onboarding Tour Overlay will solely focus on the flow of the overlay.


== Ownership ==
== Ownership ==


  Engineering lead
  Engineering lead
   [mailto:jdolske@mozilla.com Justin Dolske]
   [mailto:tchien@mozilla.com Tim Chien]


  Engineering Team
  Engineering Team
Line 49: Line 64:
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
Onboarding Tour Overlay which includes:
1. Profile Refresh<br />
2. Onboarding Tour Overlay which includes:
 
'''V1 New User'''
* Private Browsing
* Private Browsing
* Addons
* Addons
Line 56: Line 74:
* Default Browser
* Default Browser
* Sync
* Sync
'''V2 New User'''
* Performance
* Private Browsing
* Screenshots
* Add-ons
* Customize
* Default Browser
'''V2 Updated User'''
* Performance
* Library
* Screenshots
* Search
* Customize
* Sync
These features above are shown in the overlay to give the user a walk through/introduction of Firefox’s innovative features.


=== Out of Scope ===
=== Out of Scope ===
Everything not coupled with the Photon Onboarding Tour Overlay is out of scope.
Everything that does not include the features in scope, about:newtab, about:home, profile refresh, and the hamburger menu. Fennec is also out of scope.


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
  Windows 7 & 10
  Windows 7 x64: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
  Ubuntu 17.04
Windows 10 x64: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz, 3601 Mhz
  Mac 10.12
  Ubuntu 17.04 x64:Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
  Mac 10.12 x64: 2.7 GHz Intel Core i5


= Test Strategy =  
= Test Strategy =  
Line 73: Line 110:
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
|-
|-
| RAC-1 || User is irritated by not being able to quickly dismiss the onboarding overlay || TO-1 || 2-Moderate || 1-Unlikely || 1-Low || 2
| RAC-1 || User is irritated by not being able to quickly dismiss the onboarding overlay || TO-1 || 2-Moderate || 1-Unlikely || 1-Low || 2
|-
| RAC-2 || Tour shows when updating from 55 to 56 || TO-2 || 3-High || 1-Unlikely || 3-High || 9
|-
| RAC-3 || New user tour does not display with a new profile when 56 and 57 are launched || TO-2 || 3-High || 1-Unlikely || 3-High || 9
|-
|-
| RAC-2 || Tour overlay shows when updating from 55 to 56 (Tour will show for update from 56 to 57) || TO-2 || 3-High || 2-Possible || 3-High || 18
| RAC-4 || Updated User tour does not display when user updates from 56 to 57 || TO-2 || 3-High || 1-Unlikely || 3-High || 9
|-
|-
| RAC-3 || User sees the tour overlay once when updating from 55 to 56 [https://bugzilla.mozilla.org/show_bug.cgi?id=1377470 Bug 1377470] || TO-2 || 3-High || 2-Possible || 3-High || 18
| RAC-5 || Currently no automation test coverage || || 3-High || 2-Possible || 2-Medium || 12
|}
|}


Line 101: Line 142:
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
|-
|-
| 1 || Click outside of tour overlay || Tour overlay closes when user clicks out of tour overlay || The tour overlay successfully closes when the user clicks out of the tour overlay || Manual || RAC-1 || Eng Team
| 1 || Click outside of tour overlay || Tour overlay is easily dismissed/closed || The tour overlay successfully closes when the user clicks out of the tour overlay or the x in the top right corner || Manual || RAC-1 || Eng Team
|-
|-
| 2 || Updating Firefox || Updates to Firefox show the tour overlay only when user updates from a version less than 55 to 56. (Tour will show when updating from 56 to 57) || The tour overlay is not shown when the user updates from 55 to 56. The tour overlay is shown when the user updates from 54 to 56 or 56 to 57. || Manual || RAC-2, RAC-3 || Eng Team
| 2 || Updating Firefox || Updates to Firefox show the tour overlay only when user updates from a version less than 55 to 56. (Tour will show when updating from 56 to 57) || The tour overlay is not shown when the user updates from 55 to 56. The tour overlay is shown when the user updates from 54 to 56 or 56 to 57. || Manual || RAC-2, RAC-3 || Eng Team
Line 157: Line 198:
| Test case creation V2 || [https://testrail.stage.mozaws.net/index.php?/suites/view/1372&group_by=cases:section_id&group_order=asc Onboarding Test Case Creation V2]
| Test case creation V2 || [https://testrail.stage.mozaws.net/index.php?/suites/view/1372&group_by=cases:section_id&group_order=asc Onboarding Test Case Creation V2]
|-
|-
| Test case execution V2 || [https://testrail.stage.mozaws.net/index.php?/plans/view/5624 Onboarding Test Case Execution V2]
| Test case execution V2 || [https://testrail.stage.mozaws.net/index.php?/plans/view/5708 Onboarding Test Case Execution V2]
|-
|-
| Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id=1354046 Overlay Meta Bug]
| Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id=1354046 Overlay Meta Bug]
|-
| Trello || [https://trello.com/c/oMvvrfVD/284-photon-onboarding-overlay-includes-6-original-tours Overlay Trello Card]
|-
| Specs V1 || [https://mozilla.invisionapp.com/share/4MBDUMS5W#/screens/228511972_Overview Overlay V1 Specs]
|-
| Specs V2 || [https://www.google.com/url?q=https://docs.google.com/document/d/1XVowXtnAzzzyLcwdFiP6cx3l9_CNBadP9_7v17XJ40s/edit&sa=D&ust=1504820227394000&usg=AFQjCNEAJ_n8hFKCm_xt-3vtyrtrZETWGg Overlay V2 Specs]
|}
|}


Line 179: Line 226:
|-
|-
| Nightly V2  
| Nightly V2  
|style="text-align:center;"|   ||  
|style="text-align:center;"| 08/18/2017  || 20170818100226
|-
|-
| Beta V2  
| Beta V2  
Line 187: Line 234:
|style="text-align:center;"|  ||  
|style="text-align:center;"|  ||  
|}
|}
= References =
* [https://trello.com/c/oMvvrfVD/284-photon-onboarding-overlay-includes-6-original-tours Overlay Trello Card]
* [https://mozilla.invisionapp.com/share/4MBDUMS5W#/screens/228511972_Overview Overlay Specs]


= Testcases =  
= Testcases =  
Line 196: Line 239:
{| class="wikitable" style="width:80%"
{| class="wikitable" style="width:80%"
|-
|-
! Test Areas !! Covered !! Details
! Test Areas !! Covered !! Details !! Reviewed By
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | Y || have to manually type in about:newtab or about:home
|style="text-align:center;" | Y || have to manually type in about:newtab or about:home || TPE Eng Team
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | Y || enabled by default which effects majority of the users
|style="text-align:center;" | Y || enabled by default which effects majority of the users || TPE Eng Team
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | Y || not all images will be shown [https://bugzilla.mozilla.org/show_bug.cgi?id=1377439 bug 1377439]
|style="text-align:center;" | Y || not all images will be shown [https://bugzilla.mozilla.org/show_bug.cgi?id=1377439 bug 1377439] || TPE Eng Team
|-
|-
| '''UI'''  
| '''UI'''  
||  ||  
||  ||  ||
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | Y || used by majority of the users
|style="text-align:center;" | Y || used by majority of the users || TPE Eng Team
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | Y || accessibility issues are P3 [https://bugzilla.mozilla.org/show_bug.cgi?id=1377300 bug 1377300]
|style="text-align:center;" | Y || accessibility issues are P3 [https://bugzilla.mozilla.org/show_bug.cgi?id=1377300 bug 1377300] || TPE Eng Team
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | Y || verify that the overlay is shown correctly
|style="text-align:center;" | Y || verify that the overlay is shown correctly || TPE Eng Team
|-
|-
| Interaction (scroll, zoom)  
| Interaction (scroll, zoom)  
|style="text-align:center;" | Y || effects the tour overlay [https://bugzilla.mozilla.org/show_bug.cgi?id=1381010 bug 1381010]
|style="text-align:center;" | Y || effects the tour overlay [https://bugzilla.mozilla.org/show_bug.cgi?id=1381010 bug 1381010] || TPE Eng Team
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | Y || accessibility issues are P3 [https://bugzilla.mozilla.org/show_bug.cgi?id=1377300 bug 1377300]
|style="text-align:center;" | Y || accessibility issues are P3 [https://bugzilla.mozilla.org/show_bug.cgi?id=1377300 bug 1377300] || TPE Eng Team
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | Y ||  
|style="text-align:center;" | Y || Verify that the product is user friendly || TPE Eng Team
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | Y || rtl issues are P1 [https://bugzilla.mozilla.org/show_bug.cgi?id=1375024 bug 1375024]
|style="text-align:center;" | Y || rtl issues are P1 [https://bugzilla.mozilla.org/show_bug.cgi?id=1375024 bug 1375024] || TPE Eng Team
|-
|-
| '''Install/Upgrade'''  
| '''Install/Upgrade'''  
||  ||  
||  ||  ||
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | Y || [https://bugzilla.mozilla.org/show_bug.cgi?id=1377470 bug 1377470]
|style="text-align:center;" | Y || [https://bugzilla.mozilla.org/show_bug.cgi?id=1377470 bug 1377470] || TPE Eng Team
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | Y || if user in 56 is in a signed status and updates to 57, we should detect it.
|style="text-align:center;" | Y || if user in 56 is in a signed status and updates to 57, we should detect it. || TPE Eng Team
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | Y || stub installer refreshes profile on windows if the user is 2 versions lower than 57 and will not restore the session [https://bugzilla.mozilla.org/show_bug.cgi?id=1369255 bug 1369255], [https://bugzilla.mozilla.org/show_bug.cgi?id=1376558 bug 1376558]
|style="text-align:center;" | Y || stub installer refreshes profile on windows if the user is 2 versions lower than 57 and will not restore the session [https://bugzilla.mozilla.org/show_bug.cgi?id=1369255 bug 1369255], [https://bugzilla.mozilla.org/show_bug.cgi?id=1376558 bug 1376558] || TPE Eng Team
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | Y || stub installer refreshes profile on windows if the user is 2 versions lower than 57 and will not restore the session [https://bugzilla.mozilla.org/show_bug.cgi?id=1369255 bug 1369255], [https://bugzilla.mozilla.org/show_bug.cgi?id=1376558 bug 1376558]
|style="text-align:center;" | Y || stub installer refreshes profile on windows if the user is 2 versions lower than 57 and will not restore the session [https://bugzilla.mozilla.org/show_bug.cgi?id=1369255 bug 1369255], [https://bugzilla.mozilla.org/show_bug.cgi?id=1376558 bug 1376558] || TPE Eng Team
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | Y || browser.onboarding.enabled defaulted on
|style="text-align:center;" | Y || browser.onboarding.enabled defaulted on || TPE Eng Team
|-
|-
| ''' Data Monitoring '''  
| ''' Data Monitoring '''  
||  ||  
||  ||  ||
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | Y || will not be in 56 but may be in 57
|style="text-align:center;" | Y || will not be in 56 but may be in 57 || TPE Eng Team
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | Y || will not be in 56 but may be in 57
|style="text-align:center;" | Y || will not be in 56 but may be in 57 || TPE Eng Team
|-
|-
| ''' Interoperability  '''  
| ''' Interoperability  '''  
||  ||  
||  ||  ||
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | Y || Private Browsing, Sync
|style="text-align:center;" | Y || Private Browsing, Sync || TPE Eng Team
|}
|}
== Test suite ==
Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1204&group_by=cases:section_id&group_order=asc Onboarding Test Suite V1]
Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1372&group_by=cases:section_id&group_order=asc Onboarding Test Suite V2]
= Bug Work =
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=1354046 1354046 - bug summary]
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1354046 1354046] )======
<div class="mw-collapsible-content">
<bugzilla>
    {
        "blocks":[1354046],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>
</div>
</div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
<div class="mw-collapsible-content">
<bugzilla>
    {
        "blocks":[1354046],
        "resolution":"FIXED",
        "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
    }
</bugzilla>
</div>
</div>


= Sign off =
= Sign off =
Line 328: Line 337:
Win 7 & 10, Ubuntu 16.04, and Mac 10.12
Win 7 & 10, Ubuntu 16.04, and Mac 10.12
*Link for the tests run
*Link for the tests run
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/plans/view/5624 Onboarding Test Run V2]
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/plans/view/5708 Onboarding Test Run V2]


<br />
<br />
Line 349: Line 358:
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" | {{mdone|}}  ||  
|style="text-align:center;" | {{mdone|}}   
|style="text-align:center;" |  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |  {{mdone|}} ||  
| style="text-align:center;" |  {{mdone|}}  
|style="text-align:center;" |  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |  {{mdone|}} ||  
|style="text-align:center;" |  {{mdone|}}  
|style="text-align:center;" |  
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" | {{N/A|}}
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" |  ||  
|style="text-align:center;" |  {{N/A|}}
|style="text-align:center;" |  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{in progress|}}
|style="text-align:center;" |  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{in progress|}}
|style="text-align:center;" |  
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry  
|style="text-align:center;" |  
|style="text-align:center;" | {{in progress|}}
|style="text-align:center;" |
|-
|-
|  Basic/Core functionality Nightly testing
|  Basic/Core functionality Nightly testing V1
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  QA mid-Nightly Signoff  
|  QA mid-Nightly Signoff V1
|style="text-align:center;"  | {{mdone|}}   
|style="text-align:center;"  | {{mdone|}}   
|style="text-align:center;"| Email Sent
|style="text-align:center;"| Email Sent
|-
|-
|  QA Nightly - Full Testing  
|  QA Nightly - Full Testing V1
|style="text-align:center;" | {{mdone|}}   
|style="text-align:center;" | {{mdone|}}   
|style="text-align:center;" | [https://drive.google.com/file/d/0B5MxBuxQK2XiOWNpZlk5S015TlE/view?usp=sharing Test Results]
|style="text-align:center;" | [https://drive.google.com/file/d/0B5MxBuxQK2XiOWNpZlk5S015TlE/view?usp=sharing Test Results]
|-
|-
|  QA pre-Beta Signoff  
|  QA pre-Beta Signoff V1
|style="text-align:center;" | {{mdone|}}  
|style="text-align:center;" | {{mdone|}}  
|style="text-align:center;" | Email Sent
|style="text-align:center;" | Email Sent
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing V1
|style="text-align:center;" |  ||  
|style="text-align:center;" |   
|style="text-align:center;" |
|-
|  QA pre-Release Signoff V1
|style="text-align:center;" |
|style="text-align:center;" | Email to be sent
|-
|  Basic/Core functionality Nightly testing V2
|style="text-align:center;" |  {{in progress|}}
|style="text-align:center;" | 
|-
|  QA mid-Nightly Signoff V2
|style="text-align:center;"  |  {{mdone|}}
|style="text-align:center;"| Email sent
|-
|  QA Nightly - Full Testing V2
|style="text-align:center;" |
|style="text-align:center;" |
|-
|  QA pre-Beta Signoff V2
|style="text-align:center;" |
|style="text-align:center;" | Email to be sent
|-
|  QA Beta - Full Testing V2
|style="text-align:center;" | 
|style="text-align:center;" |  
|-
|-
|  QA pre-Release Signoff ||  
|  QA pre-Release Signoff V2 ||  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | Email to be sent  
|}
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
|  Lawrence Mandel / RyanVM || QA Reviewer || Product Integrity || || Email
|-
| Tim Chien || Software Engineer || Engineering || 09/06/2017 || Email
|-
| Cindy Hsiang || PM || Product Management || 09/07/2017 || Email
|}
|}
Jwilliams7
355

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1178221...1180401"