QA/SHA384 Support: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
'''Revision History'''
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
{| class="wikitable" style="width:60%"
|-
! Date !! Version !! Author !! Description
|-
| 7/20/2017 || 1.0 || Abe Masresha || Created first draft
|-
| 8/10/2017 || 1.1 || Abe Masresha || updated some sections
|-
|}
= Overview =
== Purpose ==
The purpose of this work is about adding support for stronger hash algorithms than SHA1.
== Scope ==
* Testing will focus on:
::: Stability of the browser with this work.
::: Whether browser update is applied or not
* How testing will be performed:
:::  Manual
*Automation is covered by the developer
== Ownership ==
'''Mozilla Development'''<br />
  Robert Strong <br/>
'''QA Eng. Softvision Las Vegas'''<br />
[mailto:stefan.georgiev@softvision.com Stefan Georgiev - Team Lead ]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
[mailto:jwilliams@softvision.com Justin Williams - Tester]<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />
= Testing summary =
== Scope of Testing ==
=== In Scope ===
This feature will be tested manually on Windows, Mac, and Linux operating systems.
=== Out of Scope ===
Automation will be handled by the developer<br />
::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests tests] <br>
::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/unit_base_updater tests]<br>
::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/unit_service_updater tests] <br>
::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/browser Additional tests]<br>
::: [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/chrome Additional tests]<br>
This work does not apply to Mobile devices.
== Risk Analysis ==
* Update may not be applied so that the user may not get browser updates
* May behave differently in different operating systems
* Update may not be applied successfully if the browser has addon
* May behave differently if update download is interrupted by no internet connection
= Requirements for testing =
== Environments ==
Windows 10 64bits
Windows 7 32bits
Ubuntu 16.04 32 and 64bits
MacOSX 10.12
== Channel dependent settings (configs) and environment setups ==
=== Nightly ===
Firefox Nightly 56.0a1, no special configuration is required<br>
=== Beta ===
Beta 56.0bx, no special configuration is required<br>
= Test Strategy =
== Test Objectives ==
*Verify update is applied successfully
*Verify update is applied when the browser has an addon
*Verify browser is stable and does not crash with this feature
*Verify browser does not crash while updating
*Verify Firefox is responsive if update download is interrupted by no internet connection
== Builds ==
This section should contain links for builds with the feature -
* Links for Nightly build: use the latest official Nightly build
* Links for Beta build: use the latest official beta build
== Test Execution Schedule ==
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
The following table identifies the anticipated testing period available for test execution.
Line 6: Line 96:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 5/30/2017  ||
|style="text-align:center;" | 5/30/2017  ||  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | 6/7/2017 || 8/14/2017
|style="text-align:center;" | 7/20/2017 ||  
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 6/7/2017 || 8/14/2017
|style="text-align:center;" | 7/20/2017 ||  
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" |7/31/2017 || 8/1/2017
|style="text-align:center;" | 7/20/2017 ||  
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | 8/1/2017 || 8/1/2017
|style="text-align:center;" | 8/1/2017 ||  
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
Line 25: Line 115:
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | ||  
|}
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
{| class="wikitable" style="width:50%"
|-
! Process !! Tool
|-
| Test plan creation || Mozilla wiki
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 TestRail] / Google docs
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5268 TestRail]
|-
| Bugs management || Bugzilla
|}
= Status =
== Overview ==
Landed on Nightly on: 7/31/2017<br>
Merged to Beta on: [date]
= References =
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1105689 Main bug:1105689 ] <br>
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1324498  Meta bug:1324498 ]
= Testcases =
== Test Areas ==
{| class="wikitable" style="width:80%"
|-
! Test Areas !! Covered !! Details
|-
| Private Window
|style="text-align:center;" | Yes||
|-
| Multi-Process Enabled
|style="text-align:center;" |  Yes||
|-
| Multi-process Disabled
|style="text-align:center;" | Yes ||
|-
| Theme (high contrast)
|style="text-align:center;" |  Yes||
|-
| '''Install/Upgrade'''
||  ||
|-
| Feature upgrades /Browser update
|style="text-align:center;" | Yes || Will be covered on browser updates
|-
|-
| ''' Add-ons '''
|-
| Addon API required? 
|style="text-align:center;" | Yes ||
|-
| Testing with existing/popular addons
|style="text-align:center;" | Yes || Feature functionality will be tested with popular addons
|-
| ''' Web Compatibility '''
|-
| Survey of popular sites for compatibility 
|style="text-align:center;" | Yes || Firefox will be tested for stability with popular sites
|-
| ''' Interoperability  '''
|-
| Interaction of this feature with other browser features 
|style="text-align:center;" | Yes ||  Feature will be tested for its interaction with other browser features
|}
== Test suite ==
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [ link]
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.
= Bug Work =
{| class="wikitable" style="width:50%"
|-
! Bug No !! Summary !! Status !! Firefox Version
|-
| || ||  || 56
|-
| || ||  || 56
|-
|}
= Sign off =
== Criteria ==
Checklist
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
== Results ==
'''Nightly testing'''<br />
List of OSes that will be covered by testing:  see 'Environments' section <br />
*Link for the tests run
** [https://testrail.stage.mozaws.net/index.php?/suites/view/1282&group_by=cases:section_id&group_order=desc Full Test suite]<br>
**[https://testrail.stage.mozaws.net/index.php?/plans/view/5268 Tests Runs and Results]<br>
** Daily Smoke, if needed/available
** Regression Test suite, if needed/available
<br />
'''Merge to Beta Sign-off'''<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite
== Checklist ==
{| class="wikitable" style="width:60%"
|-
! Exit Criteria !! Status !! Notes/Details
|-
|  Testing Prerequisites (specs, use cases)
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |
|-
|  Testing Infrastructure setup
|style="text-align:center;" | {{mdone|}}  ||
|-
|  Test Plan Creation
| style="text-align:center;" | {{mdone|}}  ||
|-
|  Test Cases Creation
|style="text-align:center;" | {{mdone|}}  ||
|-
|  Automation Coverage ||
|style="text-align:center;" | Covered by developer
|-
|  Performance Testing ||
|style="text-align:center;" |  Covered by developer
|-
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Basic/Core functionality Nightly testing
|style="text-align:center;" | 
|style="text-align:center;" | 
|-
|  QA mid-Nightly Signoff||
|style="text-align:center;" | Missed (test build was not ready)
|-
|  QA pre-Beta Signoff||
|style="text-align:center;"| Email to be sent
|-
|  QA Beta - Full Testing
|style="text-align:center;" |  ||
|-
|  QA pre-Release Signoff ||
|style="text-align:center;" | Email to be sent
|}
|}

Revision as of 22:15, 14 August 2017

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
7/20/2017 1.0 Abe Masresha Created first draft
8/10/2017 1.1 Abe Masresha updated some sections

Overview

Purpose

The purpose of this work is about adding support for stronger hash algorithms than SHA1.

Scope

  • Testing will focus on:
Stability of the browser with this work.
Whether browser update is applied or not
  • How testing will be performed:
Manual
  • Automation is covered by the developer

Ownership

Mozilla Development

 Robert Strong

QA Eng. Softvision Las Vegas
Stefan Georgiev - Team Lead
Abe Masresha - Test Lead
Justin Williams - Tester
Grover Wimberly IV - Tester
Kanchan Kumari - Tester

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, Mac, and Linux operating systems.

Out of Scope

Automation will be handled by the developer

tests
tests
tests
Additional tests
Additional tests

This work does not apply to Mobile devices.

Risk Analysis

  • Update may not be applied so that the user may not get browser updates
  • May behave differently in different operating systems
  • Update may not be applied successfully if the browser has addon
  • May behave differently if update download is interrupted by no internet connection

Requirements for testing

Environments

Windows 10 64bits
Windows 7 32bits
Ubuntu 16.04 32 and 64bits
MacOSX 10.12


Channel dependent settings (configs) and environment setups

Nightly

Firefox Nightly 56.0a1, no special configuration is required

Beta

Beta 56.0bx, no special configuration is required

Test Strategy

Test Objectives

  • Verify update is applied successfully
  • Verify update is applied when the browser has an addon
  • Verify browser is stable and does not crash with this feature
  • Verify browser does not crash while updating
  • Verify Firefox is responsive if update download is interrupted by no internet connection

Builds

This section should contain links for builds with the feature -

  • Links for Nightly build: use the latest official Nightly build
  • Links for Beta build: use the latest official beta build

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 5/30/2017
Study documentation/specs received from developers 7/20/2017
QA - Test plan creation 7/20/2017
QA - Test cases/Env preparation 7/20/2017
QA - Nightly Testing 8/1/2017
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail / Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Landed on Nightly on: 7/31/2017
Merged to Beta on: [date]

References

Testcases

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) Yes
Install/Upgrade
Feature upgrades /Browser update Yes Will be covered on browser updates
Add-ons
Addon API required? Yes
Testing with existing/popular addons Yes Feature functionality will be tested with popular addons
Web Compatibility
Survey of popular sites for compatibility Yes Firefox will be tested for stability with popular sites
Interoperability
Interaction of this feature with other browser features Yes Feature will be tested for its interaction with other browser features

Test suite

Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [ link]
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.

Bug Work

Bug No Summary Status Firefox Version
56
56

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing: see 'Environments' section


Merge to Beta Sign-off
List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Automation Coverage Covered by developer
Performance Testing Covered by developer
All Defects Logged
Critical/Blockers Fixed and Verified
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff Missed (test build was not ready)
QA pre-Beta Signoff Email to be sent
QA Beta - Full Testing
QA pre-Release Signoff Email to be sent
Retrieved from "https://wiki.allizom.org/index.php?title=QA/SHA384_Support&oldid=1178258"