QA/Photon Structure

From MozillaWiki
< QA
Revision as of 23:13, 3 May 2017 by Gwimberly (talk | contribs) (Created page with "= Overview = == Purpose == Quality assurance plan to ensure accessibility features with e10s enabled using Windows and Linux operating system are ready for public release. N...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Overview

Purpose

Quality assurance plan to ensure accessibility features with e10s enabled using Windows and Linux operating system are ready for public release.

Note: Despite this wiki page title, this test plan is for accessibility + e10s in general, not just touchscreen support.

Quality Criteria

Risk area Requirement Status
Screen readers There should be no significant difference in using screen readers than on non-e10s In-Progress
Touch screens Touch enabled devices, including soft keyboards, should function as well as in non-e10s (confirm a11y isn't activated for touch screen users) In-Progress
IME clients Common IME clients should function as well as in non-e10s TBD
Remote Desktop clients Browser stability issues while access the browser through RDP In-Progress
Popular sites (ie, gmail, twitter, facebook, and the other Google apps No significant regression in site correctness, stability or performance with e10s and a11y compared to a11y on non-e10s In-Progress
ARIA markup There should be no significant difference in a11y exposed interfaces for e10s vs non-e10s In-Progress
General performance Overall performance of Firefox with a11y on e10s should not be notably worse than with a11y on non-e10s In-Progress

Testing summary

Scope of Testing

In Scope

The scope of our testing is the A11y+e10s accessibility and functionality and performance of the most popular sites and most commonly used 3rd party tools.

  • Integration: Verify the integration with the current browser functionalities and UI;
  • Functionality: Basic and advanced functionality to be verified according to the existing requirements;
  • Usability: Intuitive and how users interact with the feature;
  • Performance: Reference, where applicable, observed and collected performance data.

Out of Scope

We will not be testing with less popular 3rd party tools or on obscure web sites.

Requirements for testing

Environments

Testing will be performed on following OSes:

  • Windows 10 (x64)
  • Linux - Ubuntu 16.04 (x64)

Quality Assurance Strategy

Test Items

Screen Readers

Client Free/Licensed Demo Available? Demo Expiration Purchased Copy? Owner/Location
NVDA (Win) (top priority) free TBD
Window-Eyes (Win) licensed link[1] 60 days TBD TBD
JAWS (Win) licensed - TBD Yes (Windows) Softvision USA/Las Vegas
Dolphin (Win) licensed TBD TBD TBD TBD

[1] requires filling in a contact information form

Criteria Description Metric non-e10s value e10s value Criteria Met? QA Owner
Manual testing Test cases passed # (passed -TBD) out of # (total test cases run -TBD) # (passed -TBD) out of # (total test cases run -TBD) TBD (Date status updated) SoftVision

Touch Screens

Because a11y should no longer be triggered by touch screen devices, only the first metric below should block release of a11y+e10s.

Criteria Description Metric non-e10s value e10s value Criteria Met? QA Owner
Manual testing A11y enabled? (should be false) true or false true or false TBD (Date status updated) SoftVision
Manual testing TestRail Test cases passed or Google Docs # (passed -TBD) out of 26 test cases run # (passed -TBD) out of 26 test cases run TBD (Date status updated) SoftVision

IME Clients

These are very difficult to test if one is not familiar with the language under test. Until we have a better longer term test ownership solution, we'll have to rely on community feedback and bug reports.

Language Primary IME 3rd party
Japanese Win: MS-IME, macOS: Apple Japanese IME Google Japanese Input, ATOK (not free)
Simplified Chinese Pinyin ABC
Traditional Chinese Changjie
Korean Win: MS-IME, macOS: Apple Korean IME
Criteria Description Metric non-e10s value e10s value Criteria Met? QA Owner
Manual testing Community input and bug reports # of bug reports # of bug reports TBD (Date status updated) Community

Popular Sites

Criteria Description Metric non-e10s value e10s value Criteria Met? QA Owner
Manual testing Test cases passed # (passed -TBD) out of # (total test cases run -TBD) # (passed -TBD) out of # (total test cases run -TBD) TBD (Date status updated) SoftVision


Builds

TBD

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project December 2016 -
Study documentation/specs received from developers TBD -
QA - Test plan creation 12/13/2016 -
QA - Test cases/Env preparation 12/12/2016 -
QA - Nightly Testing February/March 2017 -
QA - Aurora Testing Unknown
QA - Beta Testing Unknown
Release Date Unknown

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management Bugzilla/GitHub (mainly)

Status

Overview

  • Track the dates and build number where feature was released to Nightly
  • Track the dates and build number where feature was merged to Aurora
  • Track the dates and build number where feature was merged to Release/Beta

References

Testcases

Available on TestRail and Google Doc format TestRail Google Docs

To Be filed:

  • live regions test case (bug 1322532)
  • Browser hang up when Open View Source (IME related, bug 1318900)

Overview

  • Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) No
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) No
Interraction (scroll, zoom) Yes
Usable with a screen reader Yes e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
Help/Support
Help/support interface required No Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Yes Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing Yes Requires NVDA Installation
Affects first-run or onboarding No
Does this affect partner builds? Partner build testing No
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration No
Network proxies/autoconfig No
ESR behavior changes No
Locked preferences No
Data Monitoring
Temporary or permanent telemetry monitoring No Testing was not conducted by SV QA Eng team.
Telemetry correctness testing No Testing was not conducted by SV QA Eng team.
Server integration testing No Testing was not conducted by SV QA Eng team.
Offline and server failure testing No
Load testing No Testing was not conducted by SV QA Eng team.
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions No
Testing with existing/popular addons Yes Ensure no performance/stability regressions
Security
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites Yes
Survey of many sites for compatibility Yes
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. Yes
Coordinated testing/interop across the Firefox builds: Desktop, Android, iOS No Desktop only
Interaction of this feature with other browser features Yes

Test suite

  • Full Test suite -
  • Smoke Test suite -*

Bug Work

  • Bugzilla Meta Bugs
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1252224 -- Menus [meta] Doorhanger opening transitions are janky RESOLVED ---
1352110 -- Toolbars and Customization [meta] Create a library button to show Recent Bookmarks, Recent Downloads, History, Synced Tabs, Screenshots, and Pocket List RESOLVED ---
1352120 P1 Theme :Gijs (he/him) Put a bookmark-star button in the location bar VERIFIED Firefox 56
1352692 -- Toolbars and Customization [meta] Update navigation toolbar overflow panel for Photon NEW ---
1352693 -- Toolbars and Customization [meta] Re-work customize mode, the overflow panel and the hamburger panel NEW ---
1352695 -- Toolbars and Customization [meta] Update the hamburger panel for Photon RESOLVED ---
1352697 P5 Address Bar [meta] Add a URL-bar based page action menu NEW ---
1353421 -- General [meta] Implement sidebar updates for Photon NEW ---
1354141 P1 Toolbars and Customization Mike de Boer [:mikedeboer] Allow for panelmultiview subviews to be multiple levels deep RESOLVED Firefox 55
1354144 P1 Toolbars and Customization Mike de Boer [:mikedeboer] Allow basic keyboard navigation in panelmultiview subviews VERIFIED Firefox 55
1354146 P5 Toolbars and Customization Allow keyboard navigation using accesskeys in panelmultiview subviews REOPENED ---
1363178 P1 Toolbars and Customization Mike de Boer [:mikedeboer] Move the panelmultiview JS implementation to a separate module. RESOLVED Firefox 55
1363753 P1 Toolbars and Customization Mike de Boer [:mikedeboer] Ensure that the width of panel views do not exceed that of the main view VERIFIED Firefox 55
1363756 P4 Toolbars and Customization Remove work-around for delayed, 'jumping' panel resizing issue RESOLVED ---
1366074 -- Toolbars and Customization Make it so menu sub-panels open via hover. RESOLVED ---
1366205 P1 Toolbars and Customization Mike de Boer [:mikedeboer] Add test coverage for keyboard navigation in panelviews RESOLVED Firefox 55
1366207 P1 Toolbars and Customization :Gijs (he/him) Keep subview buttons selected after subview became visible VERIFIED Firefox 57
1366219 P5 Toolbars and Customization Secondary panels in static hamburger menu should be able to fit longer translations NEW ---
1366813 P1 Toolbars and Customization :Gijs (he/him) Add customizable flexible space item in customize mode VERIFIED Firefox 56
1366844 P1 Theme :Gijs (he/him) Update sync iconography for Photon VERIFIED Firefox 57
1367761 P1 Theme :Paolo Amadini Arrow for photon hamburger arrow panel doesn't blend in anymore VERIFIED Firefox 55
1367776 -- Toolbars and Customization :Paolo Amadini Calculate the maximum height of panelmultiview before the panel is displayed RESOLVED Firefox 55
1369095 P1 Toolbars and Customization Mike de Boer [:mikedeboer] Photon panelmultiview subviews should be scrollable VERIFIED Firefox 56
1370083 P1 Toolbars and Customization Mike de Boer [:mikedeboer] [Photon] (some?) panels have ~4px blank space at the bottom underneath the footer/content VERIFIED Firefox 56
1370584 -- Toolbars and Customization The main photon menu can be cut off if Nightly is too low on the screen RESOLVED ---
1372309 P1 General :Gijs (he/him) Flip photon structure pref on nightly to get automated test coverage as well as nightly visibility / user testing exposure RESOLVED Firefox 56
1373592 -- Toolbars and Customization Ensure the resizing architecture for Photon panels can work asynchronously VERIFIED ---
1373961 P4 Theme Update the bookmarks menu button panel/menu to match photon styling better RESOLVED ---
1374315 P1 Theme :Gijs (he/him) [Photon] overflow panel should have rounded corners (not square ones) on OS X VERIFIED Firefox 57
1374603 P1 Toolbars and Customization :Gijs (he/him) history toolbar button dropdown panel is too wide VERIFIED Firefox 56
1376152 -- Toolbars and Customization "Restart to update Nightly" item in app menu is not aligned properly RESOLVED ---
1376943 P4 Toolbars and Customization [Linux] Clicking on photon menu doesn't close it RESOLVED ---
1377139 P1 Toolbars and Customization :Gijs (he/him) [Photon] back button in menu sub panels gets stuck in hover state VERIFIED Firefox 56
1377309 -- Toolbars and Customization Hamburger and page action menu jumps a pixel when opening a sub-panel like the library or 'send to devices' view RESOLVED ---
1377791 P4 Theme Bookmarks and History subview scrollbar flickers(width changes) while scrolling by mouse NEW ---
1378427 P1 Toolbars and Customization :Gijs (he/him) Context menus flicker when opened on edit/zoom controls in the panel menu VERIFIED Firefox 56
1379387 P5 Toolbars and Customization Want keyboard shortcuts to open the hamburger and overflow menus NEW ---
1383009 P1 Toolbars and Customization :Gijs (he/him) Add flexible spaces around the URL and search bar by default and replace the bookmarks button with the library one VERIFIED Firefox 57
1385018 P2 Toolbars and Customization Flexible Spacer Shouldn't Have "Pin to Overflow Menu" Option RESOLVED ---
1385042 P1 Toolbars and Customization :Gijs (he/him) Flexible Spacer Appears in popup Windows VERIFIED Firefox 57
1387415 P1 Toolbars and Customization Sam Foster [:sfoster] (he/him) Use flexible spaces (and non-active space in the toolbar in general) as window drag areas on Windows VERIFIED Firefox 58
1388832 P1 Toolbars and Customization Sam Foster [:sfoster] (he/him) Right-clicking on the Search box within the overflow panel causes 'TypeError: parent.getAttribute is not a function' VERIFIED Firefox 57
1393411 P1 Theme :Gijs (he/him) Bookmarks and History subviews are cut off VERIFIED Firefox 57
1397447 P1 Downloads Panel :Gijs (he/him) Automatically hide/show the downloads button by default, providing an option to have it permanently visible VERIFIED Firefox 57
1399502 P3 Menus [meta] Change remaining "Options" and "Preferences" references to "Settings" (including URL) NEW ---
1400669 P4 Menus [Photon] Regression: jumping footer in library menu RESOLVED ---
1401238 P1 Menus :Gijs (he/him) Deprecate Recent Bookmarks, and Pocket List from the Bookmarks Menu. VERIFIED Firefox 57
1401991 P1 Menus Mike de Boer [:mikedeboer] The "Library" toolbarbutton opens an empty popup if the toolbarbutton was clicked with the "Library" submenu opened in Firefox's main menu. VERIFIED Firefox 57
1406892 P4 Menus Hamburger menu glitches after trying to access the Library menu from the toolbar VERIFIED ---
1407591 -- Toolbars and Customization Hamburger Menu breaks if closing it during mid-transition RESOLVED ---
1407965 P3 Menus Neil Deakin Opened "Bookmarks" menu from the Library menu glitches if Firefox window is positioned close to the bottom of the screen RESOLVED ---
1408390 -- Menus "Forget" sub-menu glitches while being opened from the Overflow panel RESOLVED ---
1409377 P1 Menus Drew Willcoxon :adw Scrollbar overlaps the keyboard shortcuts in the Hamburger menu VERIFIED Firefox 58
1410114 P3 Toolbars and Customization The "Bookmarks Toolbar" option from the customize panel is offset in full screen mode RESOLVED ---
1410666 P1 Theme Jared Wein [:jaws] (please needinfo? me) Remove unused PanelUI-popup code RESOLVED Firefox 58
1411919 P4 Menus Transition from the library menu to the available sub menus is glitching VERIFIED ---
1440875 P5 Toolbars and Customization "Show all bookmarks" should be somewhere else in the hamburger/library bookmarks dropdown/panel/doorhanger/popup/menu UNCONFIRMED ---

57 Total; 10 Open (17.54%); 22 Resolved (38.6%); 25 Verified (43.86%);



Sign off

Criteria

Check list

  • All Criteria under each section of Quality Assurance Strategy should be green.
  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Aurora testing

  • TBD on TestRail

Merge to Aurora Sign-off
List of OSes that will be covered by testing

  • Link for the tests run - Not applicable
    • Full Test suite - Not applicable

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup No
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Smoke Tests Execution
Automation Coverage
Performance Testing [IN PROGRESS]
All Defects Logged
Critical/Blockers Fixed and Verified
Daily Status Report (email/etherpad statuses/ gdoc with results)
Metrics/Telemetry N/A
QA Signoff - Nightly Release Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent

Ownership

Product contact:


User Experience contact:


Engineering contact:
Gijs Kruitbosch (IRC: gijs)
Marco Mucci (IRC: )


QA:
PM for QA team - Rares Bologa (IRC: RaresB)
QA Lead - Grover Wimberly IV (IRC: Grover-QA)
QA - Kanchan Kumari (IRC: Kanchan_QA)
QA - Justin Williams (IRC: JW_SoftvisionQA)
QA - Stefan Georgiev (IRC: StefanG_QA)
QA - Abe Masresha (IRC: Abe_LV)

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
5/3/2017 1.0 Grover Wimberly IV Created first draft