QA/Photon Preferences Search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
04/25/2017	1.0	Hani Yacoub	Created first draft

Overview

Purpose

Detail the purpose of this document. For example:

• The test scope, focus areas and objectives
• The test responsibilities
• The test strategy for the levels and types of test for this release
• The entry and exit criteria
• The basis of the test estimates
• Any risks, issues, assumptions and test dependencies
• The test schedule and major milestones
• The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the New DevTools Debugger Panel project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document what and how we shall test:

• What will be tested
• How testing will be performed

Ownership

Development Team:
Product Manager: Francis Lee
Engineer: Evan Tseng (irc: evanxd)
Engineer: Ricky Chien (irc: rickychien)

Engineering Team - Nightly Testing:
Rares Bologa (irc: RaresB) - PM for QA team
Brindusa Tot (irc: brindusat) - leading QA efforts
Hani Yacoub (irc: hani) - QA
Roxana Leitan (irc: RoxanaLeitan) - QA

Testing summary

Scope of Testing

In Scope

This feature purpose is to introduce new "search" function in Preference.

The testing effort for the New DevTools Debugger Panel will be invested on the following areas:

• functionality: basic and advanced functionality to be verified according to the existing requirements;
• usability: search functionality must be easy to use and straightforward;
• accessibility: search functionality can be used with High Contrast Theme, Keyboard only or Screen Reader;

In the scope of current testing are following critical areas:

• General search field behavior for preferences
• Query submitted
• Hover on tooltip
• No search results
• Search highlight and Firefox support positioning

Requirements for testing

Environments

Full testing will be performed on the following OSes:

• Windows 10 x64
• Mac OS X 10.12
• Ubuntu 16.04

Channel dependent settings (configs) and environment setups

Nightly

text

Beta

text

Post Beta / Release

text

Test Strategy

Risk Assessment and Coverage

ID	Description / Threat Description	Covered by Test Objective	Magnitude	Probability	Priority	Impact Score
RAC-1	Search field behavior	TO-1	1-High	2-Possible	3-High	12
RAC-2	Not all search results are highlighted	TO-2	2-Moderate	2-Possible	3-High	18
RAC-3	An indicator may not correctly point the position of the result when it is hiding in a UI component	TO-3	2-Moderate	2-Possible	3-High	18

Values:

• Magnitude: 1- Low , 2-Moderate, 3-High

• Probability: 1-Unlikely, 2-Possible, 3-Almost Certain

• Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

• An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
• An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
• An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Test Type	RAC	Owners
1	General search field behavior	Search field states and behavior should match the spec	Manual	RAC-1, RAC-2	Eng Team
2	Query submitted	All search results should be highlighted according to the spec or it should show an indicator to point the position of the result when it is hiding in a UI component	Manual	RAC-1, RAC-2	Eng Team
3	Hover on tooltip	Change opacity when a cursor hover on tooltips	Manual	RAC-1, RAC-2	Eng Team
4	No search results	Search results page is displayed, showing a message that there are no search results	Manual	RAC-1, RAC-2	Eng Team
5	Search highlight and Firefox support positioning	All search results should be highlighted according to the spec and the positioning of page element	Manual	RAC-1, RAC-2	Eng Team
6	Accessibility	Testing for accessibility includes: - testing using high Contract themes - Screen Reader validation - Keyboard-only	Manual	RAC-1, RAC-2	Eng Team

Builds

This section should contain links for builds with the feature -

• Links for Nightly builds - link
• Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail/ Google docs
Test case execution	TestRail
Bugs management	Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta

References

• Meta bug 1357285 - [meta] Preferences search
• List and links for specs

 - Preferences Search Design

• User Story Used

Testcases

Test Areas

Test Areas	Covered	Details	Reviewed by
Private Window	Yes		Francis Lee
Multi-Process Enabled	Yes		Francis Lee
Multi-process Disabled	Yes	should just check that works withour errors	Francis Lee
Theme (high contrast)	Yes		Francis Lee
UI
Mouse-only operation	Yes		Francis Lee
Keyboard-only operation	Yes		Francis Lee
Display (HiDPI)	Yes		Francis Lee
Interaction (scroll, zoom)	Yes		Francis Lee
Usable with a screen reader	Yes	e.g. with NVDA	Francis Lee
Usability and/or discoverability testing	Yes	Is this feature user friendly	Francis Lee
RTL build testing	Yes		Francis Lee
Help/Support
Help/support interface required	Yes	Make sure link to support/help page exist and is easy reachable.	Francis Lee
Support documents planned(written)	Yes	Make sure support documents are written and are correct.	Francis Lee
Install/Upgrade
Feature upgrades/downgrades data as expected	No		Francis Lee
Does sync work across upgrades	No		Francis Lee
Requires install testing	No	separate feature/application installation needed (not only Firefox)	Francis Lee
Affects first-run or onboarding	No	Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.	Francis Lee
Does this affect partner builds? Partner build testing	No	yes/no options, add comment with details about who will lead testing	Francis Lee
Enterprise		Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration	No		Francis Lee
Network proxies/autoconfig	No		Francis Lee
ESR behavior changes	No		Francis Lee
Locked preferences	No		Francis Lee
Data Monitoring
Temporary or permanent telemetry monitoring	No	List of error conditions to monitor	Francis Lee
Telemetry correctness testing	No		Francis Lee
Server integration testing	No		Francis Lee
Offline and server failure testing	No		Francis Lee
Load testing	No		Francis Lee
Add-ons		If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required?	No		Francis Lee
Comprehensive API testing	No		Francis Lee
Permissions	No		Francis Lee
Testing with existing/popular addons	No		Francis Lee
Security		Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review	No		Francis Lee
Privilege escalation testing	No		Francis Lee
Fuzzing	Mo		Francis Lee
Web Compatibility		depends on the feature
Testing against target sites	No		Francis Lee
Survey of many sites for compatibility	No		Francis Lee
Interoperability		depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.	No		Francis Lee
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS	No		Francis Lee
Interaction of this feature with other browser features	No		Francis Lee

Test suite

Full Test suite - Link to test rail - tascases should be added under Firefox Desktop project link
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.

Bug Work

Meta bug: 1357285 - bug summary

Logged bugs ( blocking 12345 )

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Target milestone
15069	P3	Networking	jefft	File type url not correctly parsed and created	VERIFIED	M13

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);

Bug fix verification

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Resolution	Target milestone
1335905	P1	Settings UI	Manotej Meka (:manotejmeka)	Add experimental search capability to about:preferences	VERIFIED	FIXED	Firefox 55
1340643	P1	UI Widgets	Evan Tseng [:evanxd]	xul:button provide the ability to get access to the textNode that displays the resulting text	VERIFIED	FIXED	mozilla55
1351867	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Show search matches in a tooltip pointing at buttons if the matched text will be revealed when clicking on the button	VERIFIED	FIXED	Firefox 56
1352481	P1	Settings UI	Evan Tseng [:evanxd]	Search should find matched strings within popups and subdialogs	VERIFIED	FIXED	Firefox 56
1353862	P1	Settings UI	Evan Tseng [:evanxd]	Performing a search then clicking on a category should remove the 'Search Results' category and clear the search	VERIFIED	FIXED	Firefox 55
1353954	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Enable the browser.preferences.search preference to turn on in-content filtering of the preferences	VERIFIED	FIXED	Firefox 56
1357130	--	Settings UI		[meta] Preferences search highlight should match the spec	RESOLVED	FIXED	Firefox 56
1357352	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Should not display the element with data-hidden-from-search=true in the Preferences search result	VERIFIED	FIXED	Firefox 55
1360491	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Add the search icon in the search input field	VERIFIED	FIXED	Firefox 55
1363062	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Fix broken data-hidden-from-search=true search result	VERIFIED	FIXED	Firefox 55
1363721	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Should highlight the sub-dialog items when do preferences search	VERIFIED	FIXED	Firefox 56
1363726	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Should make search keywords be selected after clicking search field again	VERIFIED	FIXED	Firefox 56
1363743	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Redirecting searchResult to general if search field is empty	VERIFIED	FIXED	Firefox 56
1365259	P1	Settings UI	Evan Tseng [:evanxd]	Wrong message is displayed when searching for something with no search results	VERIFIED	FIXED	Firefox 55
1365847	P1	Settings UI	Evan Tseng [:evanxd]	Search results are not correct for the "perform" keyword	VERIFIED	FIXED	Firefox 56
1366214	P1	Settings UI	Evan Tseng [:evanxd]	The result not found message should be "Sorry! There are no results in Options for " on Windows	VERIFIED	FIXED	Firefox 56
1366252	P1	Settings UI	Evan Tseng [:evanxd]	"Nightly Support" link on "Search Results" page isn't working	VERIFIED	FIXED	Firefox 56
1366633	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Saved Logins button is being left aligned	VERIFIED	FIXED	Firefox 55
1366992	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Search highlight fill should match the spec	VERIFIED	FIXED	Firefox 55
1367349	P1	Settings UI	Evan Tseng [:evanxd]	Search results matches in a tooltip pointing at buttons are still displayed even after clicking on any of the preferences category	VERIFIED	FIXED	Firefox 55
1368957	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Not all elements are highlighted when searching for an element in "about:preferences#containers"	VERIFIED	FIXED	Firefox 55
1369305	P1	Settings UI	Evan Tseng [:evanxd]	Highlight wrong items when searching the "choose" keyword	VERIFIED	FIXED	Firefox 56
1369640	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	The search icon in the search input field for Windows and Linux should match the spec	VERIFIED	FIXED	Firefox 56
1369704	P1	Preferences	Ricky Chien [:rickychien] (inactive)	Menulist items aren't highlighted	VERIFIED	FIXED	mozilla55
1370490	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Hover on search tooltips should change background opacity	VERIFIED	FIXED	Firefox 56
1370491	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Show arrow indicator in menuitem if search result exists in its corresponding content	VERIFIED	FIXED	Firefox 56
1372583	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Hide about:preferences#containers panel in search result	VERIFIED	FIXED	Firefox 56
1373535	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	confusing search preference results.	VERIFIED	FIXED	Firefox 56
1373655	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Search sign and arrow indicator aren't displayed with high contrast theme	VERIFIED	FIXED	Firefox 56
1374230	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Wrong tooltip position after resizing browser window	VERIFIED	FIXED	Firefox 56
1374588	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Labels aren't highlighted after performing a search	VERIFIED	FIXED	Firefox 56
1377009	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Search field should always float on the top	VERIFIED	FIXED	Firefox 56
1377163	P1	Settings UI	Evan Tseng [:evanxd]	Remove search result from category list	VERIFIED	FIXED	Firefox 56
1377319	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Tooltip position is wrong when searching "14"	VERIFIED	FIXED	Firefox 56
1377845	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Search tooltip is displayed in the wrong position within subdialogs	VERIFIED	FIXED	Firefox 56
1378330	P1	Themes	Ricky Chien [:rickychien] (inactive)	about:addons search field lost its submit button	VERIFIED	FIXED	mozilla56
1378359	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Wrong direction of arrow indicator is displayed on RTL Arabic build	VERIFIED	FIXED	Firefox 56
1382566	--	Settings UI	:prathiksha	Ctrl+F should focus the preferences search input	VERIFIED	FIXED	Firefox 57
1383456	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Default Search and One-Click shows up twice in preference search result	VERIFIED	FIXED	Firefox 56
1384566	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Wrong Tooltip position after searching for certain keywords	VERIFIED	FIXED	Firefox 56
1384830	P3	Settings UI	Tim Guan-tin Chien [:timdream] (please needinfo)	Preferences search is janky and unresponsive	VERIFIED	FIXED	Firefox 57
1385218	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	The position of the items which doesn't have an indicator is wrong.	VERIFIED	FIXED	Firefox 56
1385902	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	"Find in Preferences" placeholder is truncated in French	VERIFIED	FIXED	Firefox 57
1388745	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Some search results are not highlighted	VERIFIED	FIXED	Firefox 57
1390895	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Learn more label is not highlighted	VERIFIED	FIXED	Firefox 57
1396583	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Wrong Tooltip position after searching for "firefox"/"nightly"	VERIFIED	FIXED	Firefox 57
1397723	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Some strings show up as a tooltip even though the string was not found	VERIFIED	FIXED	Firefox 57
1397729	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Strings inside cookies dialog are not found when searched	VERIFIED	FIXED	Firefox 57
1397731	P1	Settings UI	Ricky Chien [:rickychien] (inactive)	Highlight is not removed from dialogs after searching for something and picking a category from Preferences	VERIFIED	FIXED	Firefox 57
1397737	P3	Settings UI	Evan Tseng [:evanxd]	Entering very long strings in the search field will result in cutting the top of the tooltip	VERIFIED	FIXED	Firefox 58
1400420	P3	Settings UI	:Gijs (he/him)	Focusing to element can be covered by browser.preferences.search header (either because moving focus upwards, or because prefs are scrolled down but focus is at the top)	RESOLVED	FIXED	97 Branch
1446657	P5	Settings UI	Jared Wein [:jaws] (please needinfo? me)	Backspacing in options search makes the window/scrollbar flicker	VERIFIED	FIXED	Firefox 61

52 Total; 0 Open (0%); 2 Resolved (3.85%); 50 Verified (96.15%);

Sign off

Criteria

Check list

• All test cases should be executed
• Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite, link to TestRail - Tests Runs and Results link
  • Daily Smoke, lif needed/available
  • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release		Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release		Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release		Email to be sent