QA/Sync Tabs Sidebar: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
 
(47 intermediate revisions by 4 users not shown)
Line 9: Line 9:
| 2/15/2016 || 1.0 || Adrian Florinescu || Created first draft
| 2/15/2016 || 1.0 || Adrian Florinescu || Created first draft
|-
|-
| || || ||  
| 2/17/2016 || 1.1 || Brindusa Tot || Complete with general details form received documentation
|}
|}


Line 25: Line 25:


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This wiki details the testing that will be performed by the project team for the Synced Tabs Sidebar project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
* What will be tested
* What will be tested
* How testing will be performed
* How testing will be performed


== Ownership ==
== Ownership ==
Developer contacts: [mailto:ewong@mozilla.com Edwin Wong]
Technical Program Manager contact: [mailto:edwong@mozilla.com Edwin Wong]


QA:<br />
Engineering contact: [mailto:kthiessen@mozilla.com Karl Thiessen ]
 
 
'''QA''':<br />
[mailto:rares.bologa@softvisioninc.eu Rares Bologa] - PM for QA team
[mailto:rares.bologa@softvisioninc.eu Rares Bologa] - PM for QA team


Line 39: Line 42:
[mailto:adrian.florinescu@softvision.ro Adrian Florinescu], QA
[mailto:adrian.florinescu@softvision.ro Adrian Florinescu], QA


[mailto:simona.marcu@softvision.ro Simona Badau], QA
[mailto:ovidiu.boca@softvision.ro Ovidiu Boca], QA
 
[mailto:liviu.cirdei@softvision.ro Liviu Cirdei], QA
 
[mailto:camelia.badau@softvision.ro Camelia Badau], QA


[mailto:ovidiu.boca@softvision.ro Ovidiu Boca], QA
= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
Line 62: Line 67:
== Environments ==
== Environments ==
Testing will be performed on following OSes:
Testing will be performed on following OSes:
* Bulleted list item
* Windows 7
* Windows 7
* Windows 10
* Windows 10
Line 87: Line 91:
  || Manual || Eng Team
  || Manual || Eng Team
|-
|-
| 2 || Sync Side bar functionality || The objective the test is trying to demonstrate that sync tab sidebar can be opened from different browser location || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Automation/ Regression/ Performance/ Usability/ Security/ Telemetry || Eng Team
| 2 || Sync Sidebar functionality || The objective of these test suites is to verify that all requirements related to the Sync Sidebar functionality are met and that they are functioning as expected || The criteria that will be evaluated to demonstrate the test is successful || Manual/ Performance/ Usability || Eng Team
|-
|-
| 3 || Repeat for each feature/sub-function || || || ||
| 3 || Sync Sidebar usability || The objective of these test suites is to verify that the application is user friendly.|| The criteria that will be evaluated to demonstrate the test is successful || Manual/ Usability || Eng Team
|-
|-
| 4 || || || || ||
| 4 || Sync Sidebar Integration  || The objective of these test suites is to ensure that no issues come from using synced tabs with different devices/OS's combinations;  || The criteria that will be evaluated to demonstrate the test is successful || Manual || Eng Team
|-
| 5 || Regression testing on the existing logged and fixed Bugzilla bugs  || Verify marked as fixed bugs against the Nightly Builds || The criteria that will be evaluated to demonstrate the test is successful ||  Manual/Regression testing || Eng Team
|}
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly builds: starting with Nightly 47: https://archive.mozilla.org/pub/firefox/nightly/2016/01/2016-01-25-06-06-32-mozilla-central/  
 
* Nightly builds with the fix are available to the the [http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ link]
* Links for Aurora builds
* Links for Aurora builds
* Links for Beta builds
* Links for Beta builds
Line 116: Line 123:
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 02/16/2016
|style="text-align:center;" | 02/16/2016
|style="text-align:center;" | 02/19/2016
|style="text-align:center;" | 02/18/2016
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
Line 127: Line 134:
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" | TBD || TBD
|style="text-align:center;" | TBD
|style="text-align:center;" | TBD
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | TBD || TBD
|style="text-align:center;" | TBD  
|style="text-align:center;" | TBD
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | TBD || TBD
|style="text-align:center;" | TBD  
|style="text-align:center;" | TBD
|}
|}


Line 153: Line 163:
= Status =  
= Status =  
== Overview ==
== Overview ==
  Track the dates and build number where feature was released to Nightly
  <small>'''<font color="green">[LANDED]</font>'''</small> Nightly 47: first landed on - '''01.25.2016'''
* https://archive.mozilla.org/pub/firefox/nightly/2016/01/2016-01-25-06-06-32-mozilla-central/
  Track the dates and build number where feature was merged to Aurora
  Track the dates and build number where feature was merged to Aurora
  Track the dates and build number where feature was merged to Release/Beta
  Track the dates and build number where feature was merged to Release/Beta


== Risk analysis ==
== Risk analysis ==
Identify the high-risk assumptions
Lack of the documentation - for this kind of feature we should receive a Release Notes document, summing up at least a basic description of what is to be expected from the feature in that release.
Identify existing bugs on the feature with high risk
 
Identify if other areas are affected by the fix
= References =
= References =
* List and links for specs
* Sync Tabs Sidebar mocks: [https://mozilla.invisionapp.com/share/KW48742BJ#/screens/120194644 link]
  List and links for available specs - documents, user stories, specifications
* Meta bug : bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1214379 1214379]
* Meta bug
 
= Testcases =  
= Testcases =  
== Overview ==
== Overview ==
Line 178: Line 184:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| '''UI'''  
| '''UI'''  
||  ||  
| style="text-align:center;" |  ||
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | N/A || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly


|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
||  ||  
| style="text-align:center;" |  ||
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | TBD || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | TBD || Make sure support documents are written and are correct.


|-
|-
| '''Install/Upgrade'''  
| '''Install/Upgrade'''  
|| ||  
 
| style="text-align:center;" | ||
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | N/A ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | N/A || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | N/A || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | N/A || yes/no options, add comment with details about who will lead testing


|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
|| ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
|style="text-align:center;" |  
||  Raise up the topic to developers to see if they are expecting to work different on ESR builds  
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" |  ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | N/A ||


|-
|-
| ''' Data Monitoring '''  
| ''' Data Monitoring '''  
|| ||  
|style="text-align:center;" | ||
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" |  || List of error conditions to monitor
|style="text-align:center;" | N/A || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | N/A ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" |  ||
|style="text-align:center;" |  Yes ||


|-
|-
| ''' Add-ons '''  
| ''' Add-ons '''  
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
|style="text-align:center;" |   
| If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" |  ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
| ''' Security '''  
| ''' Security '''  
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
|style="text-align:center;" |  ||  
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
| ''' Web Compatibility '''  
| ''' Web Compatibility '''  
|| || depends on the feature
|style="text-align:center;" |  
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
| ''' Interoperability  '''  
| ''' Interoperability  '''  
||  || depends on the feature
|style="text-align:center;" |  ||  
 
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
  Full Test suite - [https://docs.google.com/spreadsheets/d/1IWFQgS0-Wf_7MSLsgwYOyIzLMWmymNBj4tsSZm1Z9d4/edit#gid=604463486 link]
  Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
  Smoke Test suite - Link with the gdoc - if available/needed.
  Regression Test suite - Link with the gdoc - if available/needed.
  Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – meta bug
Tracking bug – bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1214379 1214379]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
Line 339: Line 349:
====== Bug fix verification ======
====== Bug fix verification ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
{| class="wikitable" style="width:50%"
{| class="wikitable" style="width:80%"
|-
|-
! Bug No !! Summary !! Status !! Firefox Verion
! Bug No !! Summary !! Status !! Firefox Verion
|-
|-
|  123 || bug summary || NEW || Nighly 45
|  1249574 || [Synced Tabs] Unchecked Tabs message is not displayed after Tab Syncing is disabled on another device and the Synced Tabs menu panel or sidebar is open on the first device. || Verified-Fixed || Nighly 47
|-
|  1250527 ||  [Sync Tabs Sidebar] Sidebar and panel "sync now" buttons should use the same terminology || Verified-Fixed || Nightly 47
|-
|-
| || || ||  
| 1241141 || Synced tabs panel height changes when it is accessed from History (Panel Menu) || Resolved-Fixed || Nightly 47
|-
|  1245728 ||  Sync Sidebar uses wrong twisties on Windows and Linux || Verified-Fixed || Nightly 47
|-
|  1250128 ||  [Sync Tabs Sidebar] Removing the search filter doesn't work the 1st time X is clicked || Verified-Fixed|| Nightly 47
|-
|  1249289 ||  "Synced Tabs" option is not visible under Sidebar || Resolved-Wontfix || Nightly 47
|-
|  1250532 ||    [Sync Tabs Sidebar] Enable Tabs for sync message not refreshed after Tab sync option disabled || Resolved-Duplicate (1249574)|| Nightly 47
|-
|  1250508 ||    [Sync Tabs Sidebar] Sync Tabs Sidebar extended options similar to Bookmarks menu || Resolved-Duplicate (1246156) || Nightly 47
|-
|  1250436 ||    [Synced Tabs] When you move Border line from Synced Tabs left or right it flicker  || Resolved-Wontfix || Nightly 47
|}
|}
</div></div>
</div></div>
Line 353: Line 377:
====== Logged bugs ======
====== Logged bugs ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Bug 111111<br/>
Bug 1249289 - "Synced Tabs" option is not visible under Sidebar<br/>
Bug 211111<br/>
Bug 1250532 - [Sync Tabs Sidebar] Enable Tabs for sync message not refreshed after Tab sync option disabled<br/>
Bug 1250436 - [Synced Tabs] When you move Border line from Synced Tabs left or right it flicker<br />
Bug 1250128 - [Sync Tabs Sidebar] Removing the search filter doesn't work the 1st time X is clicked<br />
Bug 1250508 - [Sync Tabs Sidebar] Sync Tabs Sidebar extended options similar to Bookmarks menu<br />
Bug 1252058 - [mac] "New Folder" button in the bookmarks modal dialog hangs Firefox if Sync is enabled.<br />
Bug 1249620 - [Synced Tabs] Open this site "www.dj-fotovideo.ro" after you bookmarked, the site is open in the Synced Tabs sidebar<br />
Bug 1250445 - [New Bookmark] "New folder" button becomes unavailable after 2-3 new created folders<br />
Bug 1249619 - [Sync Tabs Sidebar] User is not encouraged to sign in again after password is changed<br />
Bug 1250085 - Navigation within an already opened tab doesn't cause a new tabs record to be uploaded<br />
Bug 1250531 - [Sync] Duplicate devices when logout/login or disconnect/connect to sync account<br />
Bug 1250866 - [Synced Tabs] Not all tabs are synced if they are opened before user signs in<br />
Bug 1252054 - [Sync Tabs Sidebar] Order of devices and tabs from sidebar is not the same as order of devices and tabs from Synced Tabs panel<br />
Bug 1253613 - [Sync Tabs] Devices/tabs are lost after uncheck "Tabs" from Sync preferences<br />
Bug 1254544 - [Sync Tabs Sidebar] Right click options are missing for Sidebar Search<br />
 
 
</div>
</div>
</div>
</div>


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== NEW/ASSIGNED bugs ======
<div class="mw-collapsible-content">
[https://bugzilla.mozilla.org/show_bug.cgi?id=1249619 Bug 1249619] - [Sync Tabs Sidebar] User is not encouraged to sign in again after password is changed<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1253613 Bug 1253613] - [Sync Tabs] Devices/tabs are lost after uncheck "Tabs" from Sync preference<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1238871 Bug 1238871] - Tabs from a disconnected device are still displayed in Synced Tabs list<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1268013 Bug 1268013] - [Sync Tabs Sidebar] Selected devices flicker in Synctab sidebar<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1275857 Bug 1275857] - [Synced Tabs Sidebar] Not all synced tabs are displayed in Synced Tabs Sidebar on a second device
</div>
</div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bugs RESOLVED FIXED on Firefox 48 or Firefox 49, but not fixed on Firefox 47 ======
<div class="mw-collapsible-content">
[https://bugzilla.mozilla.org/show_bug.cgi?id=1247110 Bug 1247110] - Synced Tabs Sidebar - keyboard navigation issues<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1246156 Bug 1246156] - Few context menu options for Synced Tabs sidebar - wontfix for Firefox 47<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1250531 Bug 1250531] - [Sync] Duplicate devices when logout/login or disconnect/connect to sync account - wontfix for Firefox 47<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1247345 Bug 1247345] - Synced tab sidebar scrolls differently than bookmarks and history sidebar<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1251159 Bug 1251159] - [Sync Tabs Sidebar] search filter doesn't show the "clear filter" "X" button<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1252054 Bug 1252054] - [Sync Tabs Sidebar] Order of devices and tabs from sidebar is not the same as order of devices and tabs from Synced Tabs pane<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1254544 Bug 1254544] - [Sync Tabs Sidebar] Right click options are missing for Sidebar Search<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1239845 Bug 1239845] - Remove Fetching Synced Tabs row from Synced Tabs<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1239042 Bug 1239042] - Add minimum display time to Syncing… and spinner<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1270294 Bug 1270294] - OSX Synced Tabs spacing touch-ups<br/>
[https://bugzilla.mozilla.org/show_bug.cgi?id=1268036 Bug 1268036] - [Synced Tabs] The arrow that expand the device synced tabs is truncated on Ubuntu OS's<br/>
</div>
</div>


= Sign off =
= Sign off =
Line 369: Line 438:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing<br />
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
Line 379: Line 447:
*Link for the tests run
*Link for the tests run
** Full Test suite
** Full Test suite
'''Merge to Release Sign-off (05-26-2016)'''<br/>
List of OSes that will be covered by testing: <br />
* Windows 7 64bit
* Mac OSX 10.9.5
* Ubuntu 13.10 32bit, Ubuntu 14.04 64bit
* Full Test suite: [https://docs.google.com/spreadsheets/d/1IWFQgS0-Wf_7MSLsgwYOyIzLMWmymNBj4tsSZm1Z9d4/edit?pref=2&pli=1#gid=1506880050 link]


== Checklist ==
== Checklist ==
Line 386: Line 463:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" | {{mdone|}}    
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |  ||  
|style="text-align:center;" |  {{mdone|}} ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" | TBD ||  
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Daily Status Report (email/etherpad statuses/ gdoc with results) || ||  
|  Daily Status Report (email/etherpad statuses/ gdoc with results)  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Metrics/Telemetry||
|  Metrics/Telemetry  
|style="text-align:center;" |  
|style="text-align:center;" | N/A ||
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" | [https://wiki.mozilla.org/images/7/71/Firefox_47_Nightly_-_Synced_Tabs_Sidebar_Conditional_Sign_Off_1.pdf Conditional SignOff email] sent on 03.08.2016
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Aurora Release
|style="text-align:center;" |  
|style="text-align:center;"| Email to be sent  
|style="text-align:center;"| Email to be sent  
|-
|-
Line 429: Line 511:
|style="text-align:center;" |  ||  
|style="text-align:center;" |  ||  
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release  
|style="text-align:center;" |
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | Email to be sent  
|}
|}

Latest revision as of 07:11, 3 August 2016

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
2/15/2016 1.0 Adrian Florinescu Created first draft
2/17/2016 1.1 Brindusa Tot Complete with general details form received documentation

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the Synced Tabs Sidebar project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Technical Program Manager contact: Edwin Wong

Engineering contact: Karl Thiessen


QA:
Rares Bologa - PM for QA team

Brindusa Tot - leading QA efforts

Adrian Florinescu, QA

Ovidiu Boca, QA

Liviu Cirdei, QA

Camelia Badau, QA

Testing summary

Scope of Testing

In Scope

Sync Tabs is a project in Mozilla that aims to ensure your currently open tabs inside Firefox are available via Sync anywhere Firefox is supported. In addition to Sync Tabs, a corresponding sidebar will be added: Sync Tabs Sidebar, similar with Bookmarks/History sidebar. In the scope of our testing is the Sidebar for Sync Tabs and its functionality. The testing effort for SyncTab Sidebar will be invested on the following areas:

  • usability: the sybc tab sidebar must be easy to use and straightforward;
  • integration: devices and OS : the QA team will ensure no issues are related to synced tabs that come from different devices/OS's combinations;
  • functionality: basic and advanced functionality to be verified according to the existing requirements;

Out of Scope

Following areas/features are considered out of scope and will not be considered as testing zones to be handled in this test plan:

  • Sync
  • History?
  • Bookmarks?
  • Sync Tabs Sidebar on firefox mobile?

Requirements for testing

Environments

Testing will be performed on following OSes:

  • Windows 7
  • Windows 10
  • Windows XP
  • Ubuntu 15.04
  • Mac OS X 10

Testing will be done on iOS(iPad) and Android devices.

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Access Sync Tabs sidebar The objective the test is trying to demonstrate that Sync tabs sidebar can be opened from different browser location Sidebar can be opened from 4 location

1. The View > Sidebar menu
2. The "View Synced Tabs Sidebar" toggle in the menu item
3. The option in the Sidebar toolbar button
4. Keyboard shortcut (to be determined)

 Manual Eng Team
2 Sync Sidebar functionality The objective of these test suites is to verify that all requirements related to the Sync Sidebar functionality are met and that they are functioning as expected The criteria that will be evaluated to demonstrate the test is successful Manual/ Performance/ Usability Eng Team
3 Sync Sidebar usability The objective of these test suites is to verify that the application is user friendly. The criteria that will be evaluated to demonstrate the test is successful Manual/ Usability Eng Team
4 Sync Sidebar Integration The objective of these test suites is to ensure that no issues come from using synced tabs with different devices/OS's combinations; The criteria that will be evaluated to demonstrate the test is successful Manual Eng Team
5 Regression testing on the existing logged and fixed Bugzilla bugs Verify marked as fixed bugs against the Nightly Builds The criteria that will be evaluated to demonstrate the test is successful Manual/Regression testing Eng Team

Builds

This section should contain links for builds with the feature -

  • Nightly builds with the fix are available to the the link
  • Links for Aurora builds
  • Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 02/15/2016 03/07/2016
Study documentation/specs received from developers 02/15/2016 02/18/2016
QA - Test plan creation 02/16/2016 02/18/2016
QA - Test cases/Env preparation 02/16/2016 02/19/2016
QA - Nightly Testing 02/19/2016 03/07/2016
QA - Aurora Testing TBD TBD
QA - Beta Testing TBD TBD
Release Date TBD TBD

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs
Test case execution Google docs
Bugs management Bugzilla

Status

Overview

[LANDED] Nightly 47: first landed on - 01.25.2016
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Lack of the documentation - for this kind of feature we should receive a Release Notes document, summing up at least a basic description of what is to be expected from the feature in that release.

References

Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) N/A
UI
Mouse-only operation Yes
Keyboard-only operation No
Display (HiDPI) No
Interraction (scroll, zoom) Yes
Usable with a screen reader N/A e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
Help/Support
Help/support interface required TBD Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) TBD Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected N/A
Does sync work across upgrades Yes
Requires install testing N/A separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding N/A Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing N/A yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration N/A
Network proxies/autoconfig N/A
ESR behavior changes N/A
Locked preferences N/A
Data Monitoring
Temporary or permanent telemetry monitoring N/A List of error conditions to monitor
Telemetry correctness testing N/A
Server integration testing N/A
Offline and server failure testing N/A
Load testing Yes
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? N/A
Comprehensive API testing N/A
Permissions Yes
Testing with existing/popular addons No
Security
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility
Testing against target sites No
Survey of many sites for compatibility No
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS Yes
Interaction of this feature with other browser features No

Test suite

Full Test suite - link
Smoke Test suite - Link with the gdoc - if available/needed.
Regression Test suite - Link with the gdoc - if available/needed.

Bug Work

Tracking bug – bug 1214379

Bug fix verification
Bug No Summary Status Firefox Verion
1249574 [Synced Tabs] Unchecked Tabs message is not displayed after Tab Syncing is disabled on another device and the Synced Tabs menu panel or sidebar is open on the first device. Verified-Fixed Nighly 47
1250527 [Sync Tabs Sidebar] Sidebar and panel "sync now" buttons should use the same terminology Verified-Fixed Nightly 47
1241141 Synced tabs panel height changes when it is accessed from History (Panel Menu) Resolved-Fixed Nightly 47
1245728 Sync Sidebar uses wrong twisties on Windows and Linux Verified-Fixed Nightly 47
1250128 [Sync Tabs Sidebar] Removing the search filter doesn't work the 1st time X is clicked Verified-Fixed Nightly 47
1249289 "Synced Tabs" option is not visible under Sidebar Resolved-Wontfix Nightly 47
1250532 [Sync Tabs Sidebar] Enable Tabs for sync message not refreshed after Tab sync option disabled Resolved-Duplicate (1249574) Nightly 47
1250508 [Sync Tabs Sidebar] Sync Tabs Sidebar extended options similar to Bookmarks menu Resolved-Duplicate (1246156) Nightly 47
1250436 [Synced Tabs] When you move Border line from Synced Tabs left or right it flicker Resolved-Wontfix Nightly 47
Logged bugs

Bug 1249289 - "Synced Tabs" option is not visible under Sidebar
Bug 1250532 - [Sync Tabs Sidebar] Enable Tabs for sync message not refreshed after Tab sync option disabled
Bug 1250436 - [Synced Tabs] When you move Border line from Synced Tabs left or right it flicker
Bug 1250128 - [Sync Tabs Sidebar] Removing the search filter doesn't work the 1st time X is clicked
Bug 1250508 - [Sync Tabs Sidebar] Sync Tabs Sidebar extended options similar to Bookmarks menu
Bug 1252058 - [mac] "New Folder" button in the bookmarks modal dialog hangs Firefox if Sync is enabled.
Bug 1249620 - [Synced Tabs] Open this site "www.dj-fotovideo.ro" after you bookmarked, the site is open in the Synced Tabs sidebar
Bug 1250445 - [New Bookmark] "New folder" button becomes unavailable after 2-3 new created folders
Bug 1249619 - [Sync Tabs Sidebar] User is not encouraged to sign in again after password is changed
Bug 1250085 - Navigation within an already opened tab doesn't cause a new tabs record to be uploaded
Bug 1250531 - [Sync] Duplicate devices when logout/login or disconnect/connect to sync account
Bug 1250866 - [Synced Tabs] Not all tabs are synced if they are opened before user signs in
Bug 1252054 - [Sync Tabs Sidebar] Order of devices and tabs from sidebar is not the same as order of devices and tabs from Synced Tabs panel
Bug 1253613 - [Sync Tabs] Devices/tabs are lost after uncheck "Tabs" from Sync preferences
Bug 1254544 - [Sync Tabs Sidebar] Right click options are missing for Sidebar Search


NEW/ASSIGNED bugs

Bug 1249619 - [Sync Tabs Sidebar] User is not encouraged to sign in again after password is changed
Bug 1253613 - [Sync Tabs] Devices/tabs are lost after uncheck "Tabs" from Sync preference
Bug 1238871 - Tabs from a disconnected device are still displayed in Synced Tabs list
Bug 1268013 - [Sync Tabs Sidebar] Selected devices flicker in Synctab sidebar
Bug 1275857 - [Synced Tabs Sidebar] Not all synced tabs are displayed in Synced Tabs Sidebar on a second device

Bugs RESOLVED FIXED on Firefox 48 or Firefox 49, but not fixed on Firefox 47

Bug 1247110 - Synced Tabs Sidebar - keyboard navigation issues
Bug 1246156 - Few context menu options for Synced Tabs sidebar - wontfix for Firefox 47
Bug 1250531 - [Sync] Duplicate devices when logout/login or disconnect/connect to sync account - wontfix for Firefox 47
Bug 1247345 - Synced tab sidebar scrolls differently than bookmarks and history sidebar
Bug 1251159 - [Sync Tabs Sidebar] search filter doesn't show the "clear filter" "X" button
Bug 1252054 - [Sync Tabs Sidebar] Order of devices and tabs from sidebar is not the same as order of devices and tabs from Synced Tabs pane
Bug 1254544 - [Sync Tabs Sidebar] Right click options are missing for Sidebar Search
Bug 1239845 - Remove Fetching Synced Tabs row from Synced Tabs
Bug 1239042 - Add minimum display time to Syncing… and spinner
Bug 1270294 - OSX Synced Tabs spacing touch-ups
Bug 1268036 - [Synced Tabs] The arrow that expand the device synced tabs is truncated on Ubuntu OS's

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

  • Link for the tests run
    • Daily Smoke, use template from link
    • Full Test suite, use template from link
    • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Merge to Release Sign-off (05-26-2016)

List of OSes that will be covered by testing:

  • Windows 7 64bit
  • Mac OSX 10.9.5
  • Ubuntu 13.10 32bit, Ubuntu 14.04 64bit
  • Full Test suite: link

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage TBD
Performance Testing N/A
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Daily Status Report (email/etherpad statuses/ gdoc with results) [DONE]
Metrics/Telemetry N/A
QA Signoff - Nightly Release [DONE] Conditional SignOff email sent on 03.08.2016
QA Aurora - Full Testing N/A
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Sync_Tabs_Sidebar&oldid=1142441"