MozillaWiki

QA/LZMA Support for Updater: Difference between revisions

Page Discussion

QA/LZMA Support for Updater (view source)

Revision as of 09:54, 6 September 2017

2,044 bytes removed ,  6 September 2017
m
→‎Builds
 
(32 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
| RyanVM  || QA Reviewer ||Product Integrity || n/a||    Email
|-
|Matt Howell  || Software Engineer || Engineering || 6/14/2017 ||    Email
|-
| David Durst || EPM || Product Management || n/a ||  Email
|}
'''Revision History'''
'''Revision History'''


Line 37: Line 52:
  Matt Howell<br />
  Matt Howell<br />


'''QA Eng. Softvision Las Vegas'''<br />
'''QA Eng. Softvision Las Vegas - Nightly testing'''<br />
[mailto:stefan.georgiev@softvision.com Stefan Georgiev - Team Lead ]<br />
[mailto:amasresha@softvision.com Abe Masresha]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
 
[mailto:jwilliams@softvision.com Justin Williams - Tester]<br />
'''QA - Beta testing'''<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:petruta.rasa@softvision.ro Petruta Rasa]<br />
[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />
 
= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
This feature will be tested manually on Windows, Mac, and Linux operating systems.
This feature will be tested manually on Windows, Mac, and Linux operating systems. <br>
 
=== Out of Scope ===
Mobile devices are not affected by this work so this testing does not include them <br>


Automation is handled by the developer<br />
Automation is handled by the developer<br />
Automated tests: <br>
Automated tests: <br>
::[https://hg.mozilla.org/mozilla-central/file/tip/toolkit/mozapps/update/tests/unit_base_updater unit_base_updater]
::[https://hg.mozilla.org/mozilla-central/file/tip/toolkit/mozapps/update/tests/unit_base_updater unit_base_updater]
::[https://hg.mozilla.org/mozilla-central/file/tip/modules/libmar/tests/unit unit]
::[https://hg.mozilla.org/mozilla-central/file/tip/modules/libmar/
 
=== Out of Scope ===
Mobile devices are not affected by this work so this testing does not include them <br>


== Risk Analysis ==
tests/unit unit]
* Update may not be applied so that the user may not get browser updates
* May behave differently in different operating systems
* Update may not be applied successfully if the browser has add-on
* May behave differently if update download is interrupted by no internet connection
* The update file may not be in desired MAR file format


= Requirements for testing =
= Requirements for testing =
Line 70: Line 79:
  Ubuntu 16.04 32 and 64bits
  Ubuntu 16.04 32 and 64bits
  MacOSX 10.12
  MacOSX 10.12


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
Line 83: Line 91:
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.<br>
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.<br>


= Test Strategy =
= Test Strategy =  
= Test Strategy =  
== Risk Assessment and Coverage ==
== Risk Assessment and Coverage ==
Line 93: Line 100:
| RAC-1 ||  Update may not be applied to the browser || TO-1 ||  3-High || 2-Possible || 3-High ||18
| RAC-1 ||  Update may not be applied to the browser || TO-1 ||  3-High || 2-Possible || 3-High ||18
|-
|-
| RAC-2 || Update may not be applied successfully if the browser has add-on || TO-2 || 3-High || 2-Possible || 2-Medium || 18
| RAC-2 || Update may not be applied successfully if the browser has add-on || TO-2 || 3-High || 2-Possible || 2-Medium || 12
|-
|-
| RAC-3 || May behave differently in different operating systems || TO-3 || 2-Moderate || 1-Unlikely || 2-Medium || 12
| RAC-3 || The update file may not be in desired MAR file format || TO-3 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|-
|-
| RAC-4 || May behave differently if update download is interrupted by no internet connection || TO-4 || 2-Moderate || 1-Unlikely || 2-Medium || 12
| RAC-4 || May behave differently if update download is interrupted by no internet connection || TO-4 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|-
|-
| RAC-5 || May behave differently in different operating systems || TO-5 || 2-Moderate || 1-Unlikely || 2-Medium || 12
| RAC-5 || May behave differently in different operating systems || TO-5 || 2-Moderate || 1-Unlikely || 2-Medium || 4


|}
|}
Line 118: Line 125:
== Test Objectives ==
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.
{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners  
|-
|-
| 1 || Click outside of tour overlay || Tour overlay closes when user clicks out of tour overlay || The tour overlay successfully closes when the user clicks out of the tour overlay || Manual || RAC-1 || Eng Team
| 1 || Update the browser || Browser should update to its latest build  || update the browser || Manual || RAC-1,RAC-2,RAC-4,RAC-5 || Eng Team
|-
|-
| 2 || Updating Firefox || Updates to Firefox show the tour overlay only when user updates from a version less than 55 to 56. (Tour will show when updating from 56 to 57) || The tour overlay is not shown when the user updates from 55 to 56. The tour overlay is shown when the user updates from 54 to 56 or 56 to 57. || Manual || RAC-2, RAC-3 || Eng Team
| 2 ||Check the update file format || The update file should be in desired MAR file format  || Check if the file is in MAR format. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing steps are here] || Manual || RAC-3 || Eng Team
|}


== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
|-
! Project phase !! Start Date !! End Date
|-
| Start project
|style="text-align:center;" | 03/30/1017 || 11/14/2017
|-
| Study documentation/specs received from developers
|style="text-align:center;" | 03/30/2017 || 08/05/2017
|-
| QA - Test plan creation
|style="text-align:center;" | 06/21/2017 || 11/14/2017
|-
| QA - Test cases/Env preparation
|style="text-align:center;" | 07/14/2017 || 11/14/2017
|-
| QA - Nightly Testing V1
|style="text-align:center;" | 07/20/2017 || 07/25/2017
|-
| QA - Beta Testing V1
|style="text-align:center;" |  ||
|-
| Release Date V1
|style="text-align:center;" | ||
|-
| QA - Nightly Testing V2
|style="text-align:center;" | 08/18/2017 || 08/24/2017
|-
| QA - Beta Testing V2
|style="text-align:center;" |  ||
|-
| Release Date V2
|style="text-align:center;" | 11/14/2017 || 11/14/2017
|}
|}
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
{| class="wikitable" style="width:50%"
|-
! Process !! Tool
|-
| Test plan creation || Mozilla wiki
|-
| Test case creation V1 || [https://testrail.stage.mozaws.net/index.php?/suites/view/1204&group_by=cases:section_id&group_order=asc Onboarding Test Case Creation V1]
|-
| Test case execution V1 || [https://testrail.stage.mozaws.net/index.php?/plans/view/5064 Onboarding Test Case Execution V1]
|-
| Test case creation V2 || [https://testrail.stage.mozaws.net/index.php?/suites/view/1372&group_by=cases:section_id&group_order=asc Onboarding Test Case Creation V2]
|-
| Test case execution V2 || [https://testrail.stage.mozaws.net/index.php?/plans/view/5624 Onboarding Test Case Execution V2]
|-
| Bugs management || [https://bugzilla.mozilla.org/show_bug.cgi?id=1354046 Overlay Meta Bug]
|}
== Test Objectives ==
* Verify the update is in MAR file format. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing More information is available here]  <br>
*Verify update is applied successfully
*Verify browser is stable and does not crash with this feature
*Verify browser does not crash while updating
*Verify update is applied when the browser has an add-on
*Verify Firefox is responsive if update download is interrupted by no internet connection


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly build:  [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly build]
* Links for Nightly build:  [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly build]
* Links for Beta build: use beta build 56b3
* Links for Beta build: use beta build later than [https://archive.mozilla.org/pub/firefox/candidates/56.0b3-candidates/build1/ 56b3]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 205: Line 147:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 5/30/2017  ||   
|style="text-align:center;" | 5/30/2017  ||  9/25/2017
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
Line 214: Line 156:
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" |7/31/2017  || 8/1/2017
|style="text-align:center;" |7/31/2017  || 8/31/2017
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
Line 220: Line 162:
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 8/10/2017 || 9/8/2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | 9/19/2017 || 9/26/2017
|}
|}


Line 236: Line 178:
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc TestRail]  
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc TestRail]  
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5274 TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || [https://bugzilla.mozilla.org/ Bugzilla]
|}
|}


Line 305: Line 247:
|None in pre-beta testing cycle || ||  || 56
|None in pre-beta testing cycle || ||  || 56
|-
|-
 
|-
|None during Beta testing cycle || ||  || 56
|-
|}
|}


Line 318: Line 262:
'''Nightly testing'''<br />
'''Nightly testing'''<br />
List of OSes that will be covered by testing:  see 'Environments' section <br />
List of OSes that will be covered by testing:  see 'Environments' section <br />
*Link for the tests run
**Links:
** [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc Full Test suite]<br>
** [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc Full Test suite]<br>
**[https://testrail.stage.mozaws.net/index.php?/plans/view/5274 Runs and Results]<br>
**[https://testrail.stage.mozaws.net/index.php?/plans/view/5274 Runs and Results]<br>
Line 324: Line 268:
** Regression Test suite, No regression bugs were found <br />
** Regression Test suite, No regression bugs were found <br />


'''Merge to Beta Sign-off'''<br />
'''Preliminary status report on Beta'''<br />
List of OSes that will be covered by testing<br />
Preliminary status report analysis shows that updating from Firefox 55.0bX, updates first to 56.0b3 and then to latest beta available on the channel. Also, update.mar files are correct and have the right format. Preliminary recommendation is to "ship it". <br />
** Link for the tests run
 
** Full Test suite
'''Pre-Release Sign-off - GREEN 2017-09-06''' <br />
*List of OSes covered by testing: Windows 7 64-bit, Windows 10 64-bit, Ubuntu 16.04 64-bit and Mac OS X 10.11<br />
** Link for the tests run -  [https://testrail.stage.mozaws.net/index.php?/plans/view/5274 link]
** Regression Test suite, No regression bugs were found <br />


== Checklist ==
== Checklist ==
Line 371: Line 318:
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" |{{in progress|}}
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  QA pre-Release Signoff  
|  QA pre-Release Signoff  
|style="text-align:center;" |{{in progress|}}
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | Email sent on 9/6/2017
|}
|}
Petruta
Confirmed users
507

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1178389...1179946"