Latest revision as of 09:54, 6 September 2017

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name	Title	Department	Approval Date	Method
RyanVM	QA Reviewer	Product Integrity	n/a	Email
Robert Strong	Software Engineer	Engineering	n/a	Email
David Durst	EPM	Product Management	n/a	Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
7/20/2017	1.0	Abe Masresha	Created first draft
8/10/2017	1.1	Abe Masresha	updated links
8/14/2017	1.2	Abe Masresha	updated links and some sections

Overview

Purpose

The purpose of this work is adding support for stronger hash algorithms for signing a certificate to the update. It uses SHA384 algorithm with key size of 4096 bits.

Scope

Testing will focus on:

Stability of the browser with this work.

Whether browser update is applied or not

If SHA384 support is applied to update

If SHA384 support has key size of 4096 bits

How testing will be performed:

Manual

Ownership

Mozilla Development

 Robert Strong

QA Eng. Softvision Las Vegas - Nightly testing
Abe Masresha

QA - Beta testing
Petruta Rasa

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, Mac, and Linux operating systems.

Automation is handled by the developer

tests

Additional tests

Out of Scope

Mobile devices are not affected by this work so this testing does not include them

Requirements for testing

Environments

Windows 10 64bits
Windows 7 32bits
Ubuntu 16.04 32 and 64bits
MacOSX 10.12

Channel dependent settings (configs) and environment setups

Nightly

Firefox Nightly 56.0a1, Test environment setting is provided here
This patch was pushed to Mozilla-central on 7-31-2017, so all older Nightly should update through this update watershed.

Beta

Beta 56.0b3, Test environment setting is provided here
Feature was backed out Bug 1387231 so the new update watershed of beta will be 56b3.
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.

Test Strategy

Risk Assessment and Coverage

ID	Description / Threat Description	Covered by Test Objective	Magnitude	Probability	Priority	Impact Score
RAC-1	Update may not be applied to the browser	TO-1	3-High	2-Possible	3-High	18
RAC-2	Update may not be applied successfully if the browser has add-on	TO-2	3-High	2-Possible	3-High	18
RAC-3	SHA384 support may not be applied	TO-3	3-High	1-Unlikely	3-High	9
RAC-4	SHA384 key size of 4096 bits may not be applied	TO-4	2-Moderate	1-Unlikely	3-High	6
RAC-5	May behave differently if update download is interrupted by no internet connection	TO-5	2-Moderate	1-Unlikely	2-Medium	4
RAC-6	May behave differently in different operating systems	TO-6	2-Moderate	1-Unlikely	2-Medium	4

Values:

Magnitude: 1- Low , 2-Moderate, 3-High

Probability: 1-Unlikely, 2-Possible, 3-Almost Certain

Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

Ref	Function	Test Objective	Evaluation Criteria	Test Type	RAC	Owners
1	Update the browser	Browser should update to its latest build	update the browser	Manual	RAC-1,RAC-2,RAC-5,RAC-6	Eng Team
2	Check if SHA384 support is applied	SHA384 support should be applied to update	Check if SHA384 support is applied. steps are here	Manual	RAC-3	Eng Team
3	Check if SHA384 support has 4096 key size	Key size of 4096 bits should be used for the SHA384 support	Check if key size is 4096 bits. steps are here	Manual	RAC-4	Eng Team

Builds

This section should contain links for builds with the feature -

Links for Nightly build: Nightly build
Links for Beta build: use beta build later than 56b3

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	5/30/2017	9/25/2017
Study documentation/specs received from developers	7/20/2017	8/14/2017
QA - Test plan creation	7/20/2017	8/14/2017
QA - Test cases/Env preparation	7/20/2017	8/31/2017
QA - Nightly Testing	8/1/2017	8/1/2017
QA - Beta Testing	8/10/2017	9/8/2017
Release Date	9/19/2017	9/26/2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail
Test case execution	TestRail
Bugs management	Bugzilla

Status

Overview

Landed on Nightly on: 7/31/2017
Merged to Beta on: Beta 56b3 will have the patch. Start the testing when 56b4 is ready.

References

Testcases

Test Areas

Test Areas	Covered	Details
Private Window	Yes	user should be able to get update from private window as well
Multi-Process Enabled	Yes	user should be able to get update from e10s window
Multi-process Disabled	Yes	user should be able to get update from Non e10s window as well
Theme (high contrast)	Yes	high theme shouldnot affect the update
Install/Upgrade
Feature upgrades /Browser update	Yes	Will be covered on browser updates
Add-ons
Testing with existing/popular addons	Yes	Feature functionality will be tested with popular addons
Web Compatibility
Survey of popular sites for compatibility	Yes	Firefox will be tested for stability with popular sites
Interoperability
Interaction of this feature with other browser features	Yes	Feature will be tested for its interaction with other browser features

Test suite

Full Test suite - test rail

Bug Work

Bug No	Summary	Status	Firefox Version
None in pre-beta testing cycle			56
None during Beta testing cycle			56

Sign off

Criteria

Checklist

All test cases should be executed
Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing
List of OSes that will be covered by testing: see 'Environments' section

Links:
- Full Test suite
- Runs and Results
- Daily Smoke -not applicable to the pre-beta test cycle as the test build was landed on the last day of the pre-beta sign off
- Regression Test suite, No regression bugs were found

Preliminary status report on Beta
Preliminary status report analysis shows that updating from Firefox 55.0bX, updates first to 56.0b3 and then to latest beta available on the channel. Also, update.mar files are correct and have the right format. Preliminary recommendation is to "ship it".

Pre-Release Sign-off - GREEN 2017-09-06

List of OSes covered by testing: Windows 7 64-bit, Windows 10 64-bit, Ubuntu 16.04 64-bit and Mac OS X 10.11
- Link for the tests run - link
- Regression Test suite, No regression bugs were found

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	[DONE]
Testing Infrastructure setup	[DONE]
Test Plan Creation	[DONE]
Test Cases Creation	[DONE]
Automation Coverage	[DONE]	Covered by developer
Performance Testing	[DONE]	Covered by developer
All Defects Logged	N/A	No bugs found on pre-beta signoff
Critical/Blockers Fixed and Verified	N/A
QA mid-Nightly Signoff	[MISSED]	Missed (test build was not ready)
QA pre-Beta Signoff	[DONE]	Email sent on 8/1/2017
QA Beta - Full Testing	[DONE]
QA pre-Release Signoff	[DONE]	Email sent on 9/6/2017

@@ Line 1: / Line 1: @@
+'''Approvals Required / Received'''
+The following individuals are required to/have approved this Test Plan:
+{| class="wikitable"
+|-
+! Name !! Title !! Department !! Approval Date !! Method
+|-
+| RyanVM  || QA Reviewer ||Product Integrity || n/a||   Email
+|-
+|Robert Strong  || Software Engineer || Engineering || n/a||     Email
+|-
+| David Durst || EPM || Product Management	 || n/a ||    Email
+|}
 '''Revision History'''
@@ Line 35: / Line 50: @@
    Robert Strong <br/>
-'''QA Eng. Softvision Las Vegas'''<br />
+'''QA Eng. Softvision Las Vegas - Nightly testing'''<br />
-[mailto:stefan.georgiev@softvision.com Stefan Georgiev - Team Lead ]<br />
+[mailto:amasresha@softvision.com Abe Masresha]<br />
-[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
-[mailto:jwilliams@softvision.com Justin Williams - Tester]<br />
+'''QA - Beta testing'''<br />
-[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
+[mailto:petruta.rasa@softvision.ro Petruta Rasa]<br />
-[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />
 = Testing summary =
 == Scope of Testing ==
 === In Scope ===
-This feature will be tested manually on Windows, Mac, and Linux operating systems.
+This feature will be tested manually on Windows, Mac, and Linux operating systems.<br>
-=== Out of Scope ===
-Mobile devices are not affected by this work so this testing does not include them <br>
 Automation is handled by the developer<br />
@@ Line 57: / Line 69: @@
 ::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/chrome Additional tests]<br>
-== Risk Analysis ==
+=== Out of Scope ===
-* Update may not be applied so that the user may not get browser updates
+Mobile devices are not affected by this work so this testing does not include them <br>
-* May behave differently in different operating systems
-* Update may not be applied successfully if the browser has add-on
-* May behave differently if update download is interrupted by no internet connection
-* SHA384 support may not be applied
-* SHA384 may not have key size of 4096 bits
 = Requirements for testing =
@@ Line 75: / Line 82: @@
 === Nightly ===
-Firefox Nightly 56.0a1, no special configuration is required<br>
+Firefox Nightly 56.0a1, [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing Test environment setting is provided here]<br>
-This patch was pushed to Mozilla-central on 7-31-2017, so all Nightly updates should update through this watershed.
+This patch was pushed to Mozilla-central on 7-31-2017, so all older Nightly should update through this update watershed.
 === Beta ===
-Beta 56.0b3, no special configuration is required<br>
+Beta 56.0b3, [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing Test environment setting is provided here]<br>
-Feature was backed out [https://bugzilla.mozilla.org/show_bug.cgi?id=1387231 Bug 1387231] so the watershed for beta will be 56b3.<br>
+Feature was backed out [https://bugzilla.mozilla.org/show_bug.cgi?id=1387231 Bug 1387231] so the new update watershed of beta will be 56b3. <br>
-To test on 56b3, 56b4 is required as this is update related work.
+To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.<br>
 = Test Strategy =
+== Risk Assessment and Coverage ==
+{| class="wikitable"
+|-
+! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
+|-
+| RAC-1 ||  Update may not be applied to the browser || TO-1 ||  3-High || 2-Possible || 3-High ||18
+|-
+| RAC-2 || Update may not be applied successfully if the browser has add-on || TO-2 || 3-High || 2-Possible || 3-High || 18
+|-
+| RAC-3 || SHA384 support may not be applied || TO-3 || 3-High || 1-Unlikely ||  3-High || 9
+|-
+| RAC-4 || SHA384 key size of 4096 bits may not be applied || TO-4 || 2-Moderate || 1-Unlikely ||  3-High || 6
+|-
+| RAC-5 || May behave differently if update download is interrupted by no internet connection || TO-5 || 2-Moderate || 1-Unlikely || 2-Medium || 4
+|-
+| RAC-6 || May behave differently in different operating systems || TO-6 || 2-Moderate || 1-Unlikely || 2-Medium || 4
+|}
+'''Values:'''
+* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
+* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
+* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
+'''Impact Score Breakdown:'''
+* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
+* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
+* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
 == Test Objectives ==
-*Verify update is applied successfully
-*Verify update is applied when the browser has an addon
+{| class="wikitable"
-*Verify browser is stable and does not crash with this feature
+|-
-*Verify browser does not crash while updating
+! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners
-*Verify Firefox is responsive if update download is interrupted by no internet connection
+|-
-* Verify SHA384 support is applied. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing More information is available here] <br>
+| 1 || Update the browser || Browser should update to its latest build  || update the browser || Manual || RAC-1,RAC-2,RAC-5,RAC-6 || Eng Team
-* verify SHA384 has a key size of 4096 bits. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing More information is available here]  <br>
+|-
+| 2 ||Check if SHA384 support is applied || SHA384 support should be applied to update || Check if SHA384 support is applied. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing steps are here] || Manual || RAC-3 || Eng Team
+|-
+| 3 ||Check if SHA384 support has 4096 key size || Key size of 4096 bits should be used for the SHA384 support || Check if key size is 4096 bits. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing steps are here] || Manual || RAC-4 || Eng Team
+|}
 == Builds ==
 This section should contain links for builds with the feature -
 * Links for Nightly build:  [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly build]
-* Links for Beta build: use beta build later than 56b3
+* Links for Beta build: use beta build later than [https://archive.mozilla.org/pub/firefox/candidates/56.0b3-candidates/build1/ 56b3]
 == Test Execution Schedule ==
@@ Line 105: / Line 151: @@
 |-
 | Start project
-|style="text-align:center;" | 5/30/2017  ||
+|style="text-align:center;" | 5/30/2017  || 9/25/2017
 |-
 | Study documentation/specs received from developers
@@ Line 114: / Line 160: @@
 |-
 | QA - Test cases/Env preparation
-|style="text-align:center;" | 7/20/2017 || 8/1/2017
+|style="text-align:center;" | 7/20/2017 || 8/31/2017
 |-
 | QA - Nightly Testing
@@ Line 120: / Line 166: @@
 |-
 | QA - Beta Testing
-|style="text-align:center;" |  ||
+|style="text-align:center;" | 8/10/2017 || 9/8/2017
 |-
 | Release Date
-|style="text-align:center;" | ||
+|style="text-align:center;" | 9/19/2017 || 9/26/2017
 |}
@@ Line 134: / Line 180: @@
 | Test plan creation || Mozilla wiki
 |-
-| Test case creation || [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 TestRail] / Google docs
+| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1282&group_by=cases:section_id&group_order=desc TestRail]
 |-
 | Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 TestRail]
 |-
-| Bugs management || Bugzilla
+| Bugs management || [https://bugzilla.mozilla.org/ Bugzilla]
 |}
@@ Line 196: / Line 242: @@
 == Test suite ==
- Full Test suite - Link to test rail - test cases should be added under Firefox Desktop project [ link]
+Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1282&group_by=cases:section_id&group_order=desc test rail] <br>
- Smoke Test suite -not applicable
- Regression Test suite - not applicable
 = Bug Work =
@@ Line 206: / Line 250: @@
 |-
 |None in pre-beta testing cycle || ||  || 56
+|-
+|-
+|None during Beta testing cycle || ||  || 56
 |-
@@ Line 219: / Line 266: @@
 == Results ==
 '''Nightly testing'''<br />
 List of OSes that will be covered by testing:  see 'Environments' section <br />
-*Link for the tests run
+*Links:
 ** [https://testrail.stage.mozaws.net/index.php?/suites/view/1282&group_by=cases:section_id&group_order=desc Full Test suite]<br>
-**[https://testrail.stage.mozaws.net/index.php?/plans/view/5268 Tests Runs and Results]<br>
+**[https://testrail.stage.mozaws.net/index.php?/plans/view/5268 Runs and Results]<br>
-** Daily Smoke,not applicable to pre-beta test cycle
+** Daily Smoke -not applicable to the pre-beta test cycle as the test build was landed on the last day of the pre-beta sign off <br>
-** Regression Test suite, not applicable to pre-beta test cycle
+** Regression Test suite, No regression bugs were found <br />
-<br />
-'''Merge to Beta Sign-off'''<br />
+'''Preliminary status report on Beta'''<br />
-List of OSes that will be covered by testing<br />
+Preliminary status report analysis shows that updating from Firefox 55.0bX, updates first to 56.0b3 and then to latest beta available on the channel. Also, update.mar files are correct and have the right format. Preliminary recommendation is to "ship it".
-*Link for the tests run
-** Full Test suite
+'''Pre-Release Sign-off - GREEN 2017-09-06''' <br />
+*List of OSes covered by testing: Windows 7 64-bit, Windows 10 64-bit, Ubuntu 16.04 64-bit and Mac OS X 10.11<br />
+** Link for the tests run -  [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 link]
+** Regression Test suite, No regression bugs were found <br />
 == Checklist ==
@@ Line 275: / Line 323: @@
 |-
 |  QA Beta - Full Testing
-|style="text-align:center;" |{{in progress|}}
+|style="text-align:center;" |{{mdone|}}
 |style="text-align:center;" |
 |-
 |  QA pre-Release Signoff
-|style="text-align:center;" |{{in progress|}}
+|style="text-align:center;" |{{mdone|}}
-|style="text-align:center;" | Email to be sent
+|style="text-align:center;" | Email sent on 9/6/2017
 |}

QA/SHA384 Support: Difference between revisions

Latest revision as of 09:54, 6 September 2017

Contents

Overview

Purpose

Scope

Ownership

Testing summary

Scope of Testing

In Scope

Out of Scope

Requirements for testing

Environments

Channel dependent settings (configs) and environment setups

Nightly

Beta

Test Strategy

Risk Assessment and Coverage

Test Objectives

Builds

Test Execution Schedule

Testing Tools

Status

Overview

References

Testcases

Test Areas

Test suite

Bug Work

Sign off

Criteria

Results

Checklist

Navigation menu

QA/SHA384 Support: Difference between revisions

Latest revision as of 09:54, 6 September 2017

Overview

Purpose

Scope

Ownership

Testing summary

Scope of Testing

In Scope

Out of Scope

Requirements for testing

Environments

Channel dependent settings (configs) and environment setups

Nightly

Beta

Test Strategy

Risk Assessment and Coverage

Test Objectives

Builds

Test Execution Schedule

Testing Tools

Status

Overview

References

Testcases

Test Areas

Test suite

Bug Work

Sign off

Criteria

Results

Checklist

Navigation menu

Search