QA/SHA384 Support
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 7/20/2017 | 1.0 | Abe Masresha | Created first draft |
| 8/10/2017 | 1.1 | Abe Masresha | updated links |
| 8/14/2017 | 1.2 | Abe Masresha | updated links and some sections |
Overview
Purpose
The purpose of this work is adding support for stronger hash algorithms for signing a certificate to the update. It uses SHA384 algorithm with key size of 4096 bits.
Scope
- Testing will focus on:
- Stability of the browser with this work.
- Whether browser update is applied or not
- If SHA384 support is applied to update
- If SHA384 support has key size of 4096 bits
- How testing will be performed:
- Manual
Ownership
Mozilla Development
Robert Strong
QA Eng. Softvision Las Vegas
Stefan Georgiev - Team Lead
Abe Masresha - Test Lead
Justin Williams - Tester
Grover Wimberly IV - Tester
Kanchan Kumari - Tester
Testing summary
Scope of Testing
In Scope
This feature will be tested manually on Windows, Mac, and Linux operating systems.
Out of Scope
Mobile devices are not affected by this work so this testing does not include them
Automation is handled by the developer
Risk Analysis
- Update may not be applied so that the user may not get browser updates
- May behave differently in different operating systems
- Update may not be applied successfully if the browser has add-on
- May behave differently if update download is interrupted by no internet connection
- SHA384 support may not be applied
- SHA384 may not have key size of 4096 bits
Requirements for testing
Environments
Windows 10 64bits Windows 7 32bits Ubuntu 16.04 32 and 64bits MacOSX 10.12
Channel dependent settings (configs) and environment setups
Nightly
Firefox Nightly 56.0a1, Test environment setting is provided here
This patch was pushed to Mozilla-central on 7-31-2017, so all older Nightly should update through this update watershed.
Beta
Beta 56.0b3, Test environment setting is provided here
Feature was backed out Bug 1387231 so the new update watershed of beta will be 56b3.
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.
Test Strategy
Test Objectives
- Verify update is applied successfully
- Verify update is applied when the browser has an addon
- Verify browser is stable and does not crash with this feature
- Verify browser does not crash while updating
- Verify Firefox is responsive if update download is interrupted by no internet connection
- Verify SHA384 support is applied. More information is available here
- verify SHA384 has a key size of 4096 bits. More information is available here
Builds
This section should contain links for builds with the feature -
- Links for Nightly build: Nightly build
- Links for Beta build: use beta build later than 56b3
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 5/30/2017 | |
| Study documentation/specs received from developers | 7/20/2017 | 8/14/2017 |
| QA - Test plan creation | 7/20/2017 | 8/14/2017 |
| QA - Test cases/Env preparation | 7/20/2017 | 8/1/2017 |
| QA - Nightly Testing | 8/1/2017 | 8/1/2017 |
| QA - Beta Testing | ||
| Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail / Google docs |
| Test case execution | TestRail |
| Bugs management | Bugzilla |
Status
Overview
Landed on Nightly on: 7/31/2017
Merged to Beta on: Beta 56b3 will have the patch. Start the testing when 56b4 is ready.
References
Testcases
Test Areas
| Test Areas | Covered | Details |
|---|---|---|
| Private Window | Yes | user should be able to get update from private window as well |
| Multi-Process Enabled | Yes | user should be able to get update from e10s window |
| Multi-process Disabled | Yes | user should be able to get update from Non e10s window as well |
| Theme (high contrast) | Yes | high theme shouldnot affect the update |
| Install/Upgrade | ||
| Feature upgrades /Browser update | Yes | Will be covered on browser updates |
| Add-ons | ||
| Testing with existing/popular addons | Yes | Feature functionality will be tested with popular addons |
| Web Compatibility | ||
| Survey of popular sites for compatibility | Yes | Firefox will be tested for stability with popular sites |
| Interoperability | ||
| Interaction of this feature with other browser features | Yes | Feature will be tested for its interaction with other browser features |
Test suite
Full Test suite - Link to test rail - test cases should be added under Firefox Desktop project [ link] Smoke Test suite -not applicable Regression Test suite - not applicable
Bug Work
| Bug No | Summary | Status | Firefox Version |
|---|---|---|---|
| None in pre-beta testing cycle | 56 |
Sign off
Criteria
Checklist
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing: see 'Environments' section
- Link for the tests run
- Full Test suite
- Tests Runs and Results
- Daily Smoke,not applicable to pre-beta test cycle
- Regression Test suite, not applicable to pre-beta test cycle
- Full Test suite
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | [DONE] | |
| Testing Infrastructure setup | [DONE] | |
| Test Plan Creation | [DONE] | |
| Test Cases Creation | [DONE] | |
| Automation Coverage | [DONE] | Covered by developer |
| Performance Testing | [DONE] | Covered by developer |
| All Defects Logged | N/A | No bugs found on pre-beta signoff |
| Critical/Blockers Fixed and Verified | N/A | |
| QA mid-Nightly Signoff | [MISSED] | Missed (test build was not ready) |
| QA pre-Beta Signoff | [DONE] | Email sent on 8/1/2017 |
| QA Beta - Full Testing | IN PROGRESS | |
| QA pre-Release Signoff | IN PROGRESS | Email to be sent |