MozillaWiki

QA/SHA384 Support: Difference between revisions

Page Discussion

QA/SHA384 Support (view source)

Revision as of 09:54, 6 September 2017

4,328 bytes added ,  6 September 2017
m
→‎Builds
 
(47 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
| RyanVM  || QA Reviewer ||Product Integrity || n/a||  Email
|-
|Robert Strong  || Software Engineer || Engineering || n/a||    Email
|-
| David Durst || EPM || Product Management || n/a ||    Email
|}
'''Revision History'''
'''Revision History'''


Line 9: Line 24:
| 7/20/2017 || 1.0 || Abe Masresha || Created first draft
| 7/20/2017 || 1.0 || Abe Masresha || Created first draft
|-
|-
| 8/10/2017 || 1.1 || Abe Masresha || updated some sections
| 8/10/2017 || 1.1 || Abe Masresha || updated links
|-
| 8/14/2017 || 1.2 || Abe Masresha || updated links and some sections
|-
|-
|}
|}
Line 15: Line 32:
= Overview =
= Overview =
== Purpose ==
== Purpose ==
The purpose of this work is about adding support for stronger hash algorithms than SHA1.
The purpose of this work is adding support for stronger hash algorithms for signing a certificate to the update. It uses SHA384 algorithm with key size of 4096 bits.
 
== Scope ==
== Scope ==
* Testing will focus on:  
* Testing will focus on:  


::: Stability of the browser with this work.
::: Stability of the browser with this work.
::: Whether browser update is applied or not
::: Whether browser update is applied or not
::: If SHA384 support is applied to update
::: If SHA384 support has key size of 4096 bits


* How testing will be performed:
* How testing will be performed:
:::  Manual
:::  Manual
*Automation is covered by the developer


== Ownership ==
== Ownership ==
Line 31: Line 50:
   Robert Strong <br/>
   Robert Strong <br/>


'''QA Eng. Softvision Las Vegas'''<br />
'''QA Eng. Softvision Las Vegas - Nightly testing'''<br />
[mailto:stefan.georgiev@softvision.com Stefan Georgiev - Team Lead ]<br />
[mailto:amasresha@softvision.com Abe Masresha]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
 
[mailto:jwilliams@softvision.com Justin Williams - Tester]<br />
'''QA - Beta testing'''<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:petruta.rasa@softvision.ro Petruta Rasa]<br />
[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />


= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
This feature will be tested manually on Windows, Mac, and Linux operating systems.
This feature will be tested manually on Windows, Mac, and Linux operating systems.<br>


=== Out of Scope ===
Automation is handled by the developer<br />
Automation will be handled by the developer<br />


::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests tests] <br>
::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests tests] <br>
Line 52: Line 69:
::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/chrome Additional tests]<br>
::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/chrome Additional tests]<br>


This work does not apply to Mobile devices.
=== Out of Scope ===
 
Mobile devices are not affected by this work so this testing does not include them <br>
== Risk Analysis ==
* Update may not be applied so that the user may not get browser updates
* May behave differently in different operating systems
* Update may not be applied successfully if the browser has addon
* May behave differently if update download is interrupted by no internet connection


= Requirements for testing =
= Requirements for testing =
Line 66: Line 78:
  Ubuntu 16.04 32 and 64bits
  Ubuntu 16.04 32 and 64bits
  MacOSX 10.12
  MacOSX 10.12


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==


=== Nightly ===
=== Nightly ===
Firefox Nightly 56.0a1, no special configuration is required<br>
Firefox Nightly 56.0a1, [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing Test environment setting is provided here]<br>
This patch was pushed to Mozilla-central on 7-31-2017, so all older Nightly should update through this update watershed.


=== Beta ===
=== Beta ===
Beta 56.0bx, no special configuration is required<br>
Beta 56.0b3, [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing Test environment setting is provided here]<br>
Feature was backed out [https://bugzilla.mozilla.org/show_bug.cgi?id=1387231 Bug 1387231] so the new update watershed of beta will be 56b3. <br>
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.<br>


= Test Strategy =  
= Test Strategy =  
== Risk Assessment and Coverage ==
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 ||  Update may not be applied to the browser || TO-1 ||  3-High || 2-Possible || 3-High ||18
|-
| RAC-2 || Update may not be applied successfully if the browser has add-on || TO-2 || 3-High || 2-Possible || 3-High || 18
|-
| RAC-3 || SHA384 support may not be applied || TO-3 || 3-High || 1-Unlikely ||  3-High || 9
|-
| RAC-4 || SHA384 key size of 4096 bits may not be applied || TO-4 || 2-Moderate || 1-Unlikely ||  3-High || 6
|-
| RAC-5 || May behave differently if update download is interrupted by no internet connection || TO-5 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|-
| RAC-6 || May behave differently in different operating systems || TO-6 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|}
'''Values:'''
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
== Test Objectives ==
== Test Objectives ==
*Verify update is applied successfully
 
*Verify update is applied when the browser has an addon
{| class="wikitable"
*Verify browser is stable and does not crash with this feature
|-
*Verify browser does not crash while updating
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners
*Verify Firefox is responsive if update download is interrupted by no internet connection
|-
| 1 || Update the browser || Browser should update to its latest build  || update the browser || Manual || RAC-1,RAC-2,RAC-5,RAC-6 || Eng Team
|-
| 2 ||Check if SHA384 support is applied || SHA384 support should be applied to update || Check if SHA384 support is applied. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing steps are here] || Manual || RAC-3 || Eng Team
|-
| 3 ||Check if SHA384 support has 4096 key size || Key size of 4096 bits should be used for the SHA384 support || Check if key size is 4096 bits. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing steps are here] || Manual || RAC-4 || Eng Team
 
|}
 


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly build: use the latest official Nightly build
* Links for Nightly build: [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly build]
* Links for Beta build: use the latest official beta build
* Links for Beta build: use beta build later than [https://archive.mozilla.org/pub/firefox/candidates/56.0b3-candidates/build1/ 56b3]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 96: Line 151:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 5/30/2017  ||  
|style="text-align:center;" | 5/30/2017  || 9/25/2017
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | 7/20/2017 ||  
|style="text-align:center;" | 7/20/2017 || 8/14/2017
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 7/20/2017 ||  
|style="text-align:center;" | 7/20/2017 || 8/14/2017
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 7/20/2017 ||  
|style="text-align:center;" | 7/20/2017 || 8/31/2017
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | 8/1/2017 ||  
|style="text-align:center;" | 8/1/2017 || 8/1/2017
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 8/10/2017 || 9/8/2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | 9/19/2017 || 9/26/2017
|}
|}


Line 125: Line 180:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 TestRail] / Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1282&group_by=cases:section_id&group_order=desc TestRail]
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || [https://bugzilla.mozilla.org/ Bugzilla]
|}
|}


= Status =  
= Status =  
== Overview ==
== Overview ==
Landed on Nightly on: 7/31/2017<br>
Landed on Nightly on: 7/31/2017 <br>
Merged to Beta on: [date]
Merged to Beta on: Beta 56b3 will have the patch. Start the testing when 56b4 is ready.


= References =
= References =
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1105689 Main bug:1105689 ] <br>
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1105689 Main bug:1105689 ] <br>
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1324498  Meta bug:1324498 ]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1324498  Meta bug:1324498 ]
* [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing Additional information to QE]


= Testcases =  
= Testcases =  
Line 148: Line 204:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | Yes||  
|style="text-align:center;" | Yes|| user should be able to get update from private window as well
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" |  Yes||  
|style="text-align:center;" |  Yes|| user should be able to get update from e10s window
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | Yes ||  
|style="text-align:center;" | Yes || user should be able to get update from Non e10s window as well
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" |  Yes||  
|style="text-align:center;" |  Yes|| high theme shouldnot affect the update
|-
|-
| '''Install/Upgrade'''  
| '''Install/Upgrade'''  
Line 168: Line 224:
| ''' Add-ons '''  
| ''' Add-ons '''  


|-
| Addon API required? 
|style="text-align:center;" | Yes ||
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
Line 189: Line 242:


== Test suite ==
== Test suite ==
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [ link]
Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1282&group_by=cases:section_id&group_order=desc test rail] <br>
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.


= Bug Work =
= Bug Work =
Line 198: Line 249:
! Bug No !! Summary !! Status !! Firefox Version
! Bug No !! Summary !! Status !! Firefox Version
|-
|-
| || ||  || 56
|None in pre-beta testing cycle || ||  || 56
|-
|-
|-
| || ||  || 56
|None during Beta testing cycle || ||  || 56
|-
|-
|}
|}


Line 213: Line 266:
== Results ==
== Results ==
'''Nightly testing'''<br />
'''Nightly testing'''<br />
List of OSes that will be covered by testing:  see 'Environments' section <br />
List of OSes that will be covered by testing:  see 'Environments' section <br />
*Link for the tests run
*Links:
** [https://testrail.stage.mozaws.net/index.php?/suites/view/1282&group_by=cases:section_id&group_order=desc Full Test suite]<br>
** [https://testrail.stage.mozaws.net/index.php?/suites/view/1282&group_by=cases:section_id&group_order=desc Full Test suite]<br>
**[https://testrail.stage.mozaws.net/index.php?/plans/view/5268 Tests Runs and Results]<br>
**[https://testrail.stage.mozaws.net/index.php?/plans/view/5268 Runs and Results]<br>
** Daily Smoke, if needed/available
** Daily Smoke -not applicable to the pre-beta test cycle as the test build was landed on the last day of the pre-beta sign off <br>
** Regression Test suite, if needed/available
** Regression Test suite, No regression bugs were found <br />
<br />
 
'''Preliminary status report on Beta'''<br />
Preliminary status report analysis shows that updating from Firefox 55.0bX, updates first to 56.0b3 and then to latest beta available on the channel. Also, update.mar files are correct and have the right format. Preliminary recommendation is to "ship it".


'''Merge to Beta Sign-off'''<br />
'''Pre-Release Sign-off - GREEN 2017-09-06''' <br />
List of OSes that will be covered by testing<br />
*List of OSes covered by testing: Windows 7 64-bit, Windows 10 64-bit, Ubuntu 16.04 64-bit and Mac OS X 10.11<br />
*Link for the tests run
** Link for the tests run -  [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 link]
** Full Test suite
** Regression Test suite, No regression bugs were found <br />


== Checklist ==
== Checklist ==
Line 245: Line 299:
|style="text-align:center;" | {{mdone|}}  ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" | Covered by developer  
|style="text-align:center;" | Covered by developer  
|-
|-
|  Performance Testing ||  
|  Performance Testing  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" |  Covered by developer  
|style="text-align:center;" |  Covered by developer  
|-
|-
|  All Defects Logged || ||
|  All Defects Logged   
|-
|style="text-align:center;" |{{N/A|}}
|  Critical/Blockers Fixed and Verified || ||
|style="text-align:center;" | No bugs found on pre-beta signoff
|-
| Basic/Core functionality Nightly testing
|style="text-align:center;" |
|style="text-align:center;" |
|-
|-
|  QA mid-Nightly Signoff||  
|  Critical/Blockers Fixed and Verified
|style="text-align:center;" | Missed (test build was not ready)
|style="text-align:center;" |{{N/A|}}||
|-
|  QA mid-Nightly Signoff
|style="text-align:center;" |{{Missed |}}
|style="text-align:center;"| Missed (test build was not ready)
|-
|-
|  QA pre-Beta Signoff||  
|  QA pre-Beta Signoff
|style="text-align:center;"| Email to be sent  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;"| Email sent on 8/1/2017
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" | 
|-
|-
|  QA pre-Release Signoff ||  
|  QA pre-Release Signoff  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" | Email sent on 9/6/2017
|}
|}
Petruta
Confirmed users
507

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1177919...1179947"