MozillaWiki

QA/LZMA Support for Updater: Difference between revisions

Page Discussion

QA/LZMA Support for Updater (view source)

Revision as of 09:54, 6 September 2017

2,431 bytes added ,  6 September 2017
m
→‎Builds
 
(35 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
| RyanVM  || QA Reviewer ||Product Integrity || n/a||    Email
|-
|Matt Howell  || Software Engineer || Engineering || 6/14/2017 ||    Email
|-
| David Durst || EPM || Product Management || n/a ||  Email
|}
'''Revision History'''
'''Revision History'''


Line 37: Line 52:
  Matt Howell<br />
  Matt Howell<br />


'''QA Eng. Softvision Las Vegas'''<br />
'''QA Eng. Softvision Las Vegas - Nightly testing'''<br />
[mailto:stefan.georgiev@softvision.com Stefan Georgiev - Team Lead ]<br />
[mailto:amasresha@softvision.com Abe Masresha]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
 
[mailto:jwilliams@softvision.com Justin Williams - Tester]<br />
'''QA - Beta testing'''<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:petruta.rasa@softvision.ro Petruta Rasa]<br />
[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />
 
= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
This feature will be tested manually on Windows, Mac, and Linux operating systems.
This feature will be tested manually on Windows, Mac, and Linux operating systems. <br>
 
=== Out of Scope ===
Mobile devices are not affected by this work so this testing does not include them <br>


Automation is handled by the developer<br />
Automation is handled by the developer<br />
Automated tests: <br>
Automated tests: <br>
::[https://hg.mozilla.org/mozilla-central/file/tip/toolkit/mozapps/update/tests/unit_base_updater unit_base_updater]
::[https://hg.mozilla.org/mozilla-central/file/tip/toolkit/mozapps/update/tests/unit_base_updater unit_base_updater]
::[https://hg.mozilla.org/mozilla-central/file/tip/modules/libmar/tests/unit unit]
::[https://hg.mozilla.org/mozilla-central/file/tip/modules/libmar/
 
=== Out of Scope ===
Mobile devices are not affected by this work so this testing does not include them <br>


== Risk Analysis ==
tests/unit unit]
* Update may not be applied so that the user may not get browser updates
* May behave differently in different operating systems
* Update may not be applied successfully if the browser has add-on
* May behave differently if update download is interrupted by no internet connection
* The update file may not be in desired MAR file format


= Requirements for testing =
= Requirements for testing =
Line 70: Line 79:
  Ubuntu 16.04 32 and 64bits
  Ubuntu 16.04 32 and 64bits
  MacOSX 10.12
  MacOSX 10.12


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
Line 84: Line 92:


= Test Strategy =  
= Test Strategy =  
== Risk Assessment and Coverage ==
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 ||  Update may not be applied to the browser || TO-1 ||  3-High || 2-Possible || 3-High ||18
|-
| RAC-2 || Update may not be applied successfully if the browser has add-on || TO-2 || 3-High || 2-Possible || 2-Medium || 12
|-
| RAC-3 || The update file may not be in desired MAR file format || TO-3 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|-
| RAC-4 || May behave differently if update download is interrupted by no internet connection || TO-4 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|-
| RAC-5 || May behave differently in different operating systems || TO-5 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
== Test Objectives ==
== Test Objectives ==
* Verify the update is in MAR file format. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing More information is available here] <br>
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
*Verify update is applied successfully
{| class="wikitable"
*Verify browser is stable and does not crash with this feature
|-
*Verify browser does not crash while updating
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners
*Verify update is applied when the browser has an add-on
|-
*Verify Firefox is responsive if update download is interrupted by no internet connection
| 1 || Update the browser || Browser should update to its latest build  || update the browser || Manual || RAC-1,RAC-2,RAC-4,RAC-5 || Eng Team
|-
| 2 ||Check the update file format || The update file should be in desired MAR file format  || Check if the file is in MAR format. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing steps are here] || Manual || RAC-3 || Eng Team
 
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly build:  [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly build]
* Links for Nightly build:  [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly build]
* Links for Beta build: use beta build 56b3
* Links for Beta build: use beta build later than [https://archive.mozilla.org/pub/firefox/candidates/56.0b3-candidates/build1/ 56b3]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 104: Line 147:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 5/30/2017  ||   
|style="text-align:center;" | 5/30/2017  ||  9/25/2017
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
Line 113: Line 156:
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" |7/31/2017  || 8/1/2017
|style="text-align:center;" |7/31/2017  || 8/31/2017
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
Line 119: Line 162:
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 8/10/2017 || 9/8/2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | 9/19/2017 || 9/26/2017
|}
|}


Line 135: Line 178:
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc TestRail]  
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc TestRail]  
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5274 TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || [https://bugzilla.mozilla.org/ Bugzilla]
|}
|}


Line 204: Line 247:
|None in pre-beta testing cycle || ||  || 56
|None in pre-beta testing cycle || ||  || 56
|-
|-
 
|-
|None during Beta testing cycle || ||  || 56
|-
|}
|}


Line 216: Line 261:
== Results ==
== Results ==
'''Nightly testing'''<br />
'''Nightly testing'''<br />
List of OSes that will be covered by testing:  see 'Environments' section <br />
List of OSes that will be covered by testing:  see 'Environments' section <br />
*Link for the tests run
**Links:
** [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc Full Test suite]<br>
** [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc Full Test suite]<br>
**[https://testrail.stage.mozaws.net/index.php?/plans/view/5274 Runs and Results]<br>
**[https://testrail.stage.mozaws.net/index.php?/plans/view/5274 Runs and Results]<br>
** Daily Smoke,not applicable to pre-beta test cycle
** Daily Smoke -not applicable to the pre-beta test cycle as the test build was landed on the last day of the pre-beta sign off <br>
** Regression Test suite, not applicable to pre-beta test cycle
** Regression Test suite, No regression bugs were found <br />
<br />


'''Merge to Beta Sign-off'''<br />
'''Preliminary status report on Beta'''<br />
List of OSes that will be covered by testing<br />
Preliminary status report analysis shows that updating from Firefox 55.0bX, updates first to 56.0b3 and then to latest beta available on the channel. Also, update.mar files are correct and have the right format. Preliminary recommendation is to "ship it". <br />
** Link for the tests run
 
** Full Test suite
'''Pre-Release Sign-off - GREEN 2017-09-06''' <br />
*List of OSes covered by testing: Windows 7 64-bit, Windows 10 64-bit, Ubuntu 16.04 64-bit and Mac OS X 10.11<br />
** Link for the tests run -  [https://testrail.stage.mozaws.net/index.php?/plans/view/5274 link]
** Regression Test suite, No regression bugs were found <br />


== Checklist ==
== Checklist ==
Line 272: Line 318:
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" |{{in progress|}}
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  QA pre-Release Signoff  
|  QA pre-Release Signoff  
|style="text-align:center;" |{{in progress|}}
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | Email sent on 9/6/2017
|}
|}
Petruta
Confirmed users
507

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1178353...1179946"