QA/Push Notifications

Test Plan (Overview)

SUMMARY

• The purpose of this wiki is to serve as a general test plan for verifying desktop Push notifications.

OWNERSHIP

• Manual tests will be defined / outlined and executed by the Cloud Services QA Team
• Manual testing will be handed off to Softvision QA Team
• Automated E2E smoke tests will be created by the Cloud Services QA Team
• Any further feature automation is TBD (by Firefox Quality Engineering Team)

TEST RESULTS

• Test results will be posted in Bugzilla (component: TBD).

SIGN-OFF CRITERIA Cloud Services QA team will verify that:

• Desktop push notifications are working properly in FF Nightly.
• Interaction between push-service (auto-push[3]), Service Workers[1], and desktop UI is working as expected.

Test Objectives

Following table contains main features to be tested:

Ref	Function	Test Objective	Evaluation Criteria
1	Notification - basic scenarios	Pop-up notifications display or not based on permissions and settings	Notifications display or disable accordingly.
2	Multi-Process (e10) Enable & Disabled, Private Window	Validate that Notifications display for all window types	Multi-Process (e10) and Private windows display notifications

Test Types

Type of Testing	Required (Yes/No)	Owners	Details/Comments
Manual	Yes	Moz QA Eng Team LV
Automation	No
Performance	No
Security	No
Usability	No
Metrics/Telemetry	No

Builds

This section should contain links for builds with the feature -

Nightly builds with the search refactor fix can be reached at link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	10.23.2015
Study documentation/specs received from developers	10.23.2015	-
QA - Test plan creation	10.23.2015
QA - Test cases/Env preparation	10.23.2015	-
QA - Nightly Testing	10.23.2015	12.16.2015
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	Google docs
Test case execution	Google docs
Bugs management	Bugzilla

Test cases

Overview

Platform(s) Windows 10, Windows 8, Windows 8.1, Windows Vista, Windows 7, Ubuntu 14.04, Mac OS X 10

Daily Smoke Test will test the following: [Cycle>Check Enable multi-process Nightly, Unchecked Enable multi-process, Nightly Private Window]

Test Name	Description
Notification Dialog	Verify that trying to display a notification causes a dialog to appear.
Notification Permissions	Verify that Permissions can be set & adhered to [Always Ask, Allow, Block] .
Notification: Timeout	Verify that a notification is auto-closed after X seconds.
Notification: Blocking	Verify that blocking notifications prevents notifications from displaying.
Notification: Not Now	Verify clicking "Not Now" from the notification permissions door-hanger causes the permissions popup to appear again.
Notification: Close Button	Verify notification can be dismissed by clicking the close button.
Notification: Redirection	Verify clicking on a notification opens a specified URL.
Notification: Pop Up Multi Message	Verify clicking the pop Notification button causes the notification to display multiple times.

Test Areas

Test Areas	Covered	Details
Private Window	[DONE]
Multi-Process Enabled	[DONE]
Multi-process Disabled	[DONE]
Display (HiDPI)	N/A
Interaction (scroll, zoom)	N/A
Theme (high contrast)	N/A

Test suites

Full Test Suite - can be reached at link

Bugs Work

Bug information can be displayed at link

Sign off

Criteria

• All test cases should be executed
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing 45.0a1

• [Complete Push 1.0] Full Test Suite results [1]
  

Beta testing 44.0b4

• [Complete Push 1.0] Full Test Suite results [2]
  

• • Manual testing covered Windows 7 x64, Windows 8 x64, Windows 8.1 x64 Windows 10 x64, Windows Vista x32, Ubuntu 14.04 x32, Mac OS x 10.11

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	[DONE]
Testing Infrastructure setup	[DONE]
Test Plan Creation	[DONE]
Test Cases Creation	[DONE]
Full Functional Tests Execution	[DONE]
Performance Testing	N/A
All Defects Logged	[DONE]
Critical/Blockers Fixed and Verified	[DONE]
Daily Status Report (email/etherpad statuses/ gdoc with results)	[DONE]
Metrics/Telemetry	N/A
UAT Test Cases (signoff By Users)	N/A
QA Signoff - Nightly Release	[DONE]	Email sent Jan 04, 2016 [Jan 06, 2016]
QA Postmortem	[DONE]	Meeting Notes

REFERENCE

• [1] http://www.w3.org/TR/service-workers/
• [2] http://www.w3.org/TR/push-api/
• [3] autopush: https://github.com/mozilla-services/autopush/
• [4] push-service: https://github.com/mozilla-services/push-service

Test Cases

NOTIFICATIONS API

Common

Shared steps between Notifications tests:

setup()

1. Open https://pdehaan.github.io/push-notification-test/
2. Click pop Notification button.

teardown()

1. Click on lock icon in the address bar.
2. Set Show Notifications permission to "Always Ask".

Weekly: In-Depth & Exploratory Tests

Notes: Manual/Automation: Test will be conducted manually Browser Window: Test will be conducted in standard browser window and Private browser windows Platforms: 32bit/64 bit Windows 7, Vista, 8, 8.1, 10; Ubuntu 14.04 (x64) and Mac OS X 10.9.5.

Title	Description
Doorhanger default state test	Verify the doorhanger reports if the site has requested any notification permissions. 1. Click the lock icon in the address bar. 2. Verify the Verify the doorhanger reports if the site has requested any notification permissions.
Pop-up Dialog	Verify that trying to display a notification causes a dialog to appear. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Click the "close" button in the upper right corner of the notifications doorhanger. 3. Click "pop Notification" button. 4. Verify the notification doorhanger appears again and requests permissions.
Close Dialog by External Means / Non-Close Button	Verify clicking outside of a notification dismisses it automatically (OS X only). 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Click outside of the notifications doorhanger. 3. Verify the notifications doorhanger is dismissed.
Learn More Link	Verify clicking on the "Learn more" link opens the target page. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Click the "Learn more..." link in the notifications doorhanger. 3. Verify the browser redirects to https://www.mozilla.org/en-US/firefox/push/ (in a new tab).
Auto Close (Timeout)	Verify that a notification is auto-closed after X seconds. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Verify the notification appears and auto-dismisses after X seconds. 4. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Blocking Notifications	Verify that blocking notifications prevents notifications from displaying. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Block Notifications" from the notifications doorhanger. 3. Verify no notification appears. 4. Click "pop Notification" button. 5. Verify no notification appears. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Not Now	Verify clicking "Not Now" from the notification permissions doorhanger causes the permissions popup to appear again. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Not Now" from the notifications doorhanger. 3. Verify no notification appears. 4. Click "pop Notification" button. 5. Verify notification doorhanger appears again. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Permission Changes	Verify changing notification permissions in the address bar causes the notification permissions screen to reappear. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Verify notification appears and auto-dismisses after X seconds. 4. Click the lock button in the address bar. 5. Change notification permissions to "Always Ask". 6. Click "pop Notification" button. 7. Verify notification doorhanger appears again. 8. Select "Always Show Notifications" from the notifications doorhanger. 9. Verify notification appears. 10. Click the lock button in the address bar. 11. Change notification permissions to "Block". 12. Click "pop Notification" button. 13. Verify no notification appears. 14. Click the lock button in the address bar. 15. Change notification permissions to "Allow". 16. Click "pop Notification" button appears. 17. Verify notification appears. 18. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Programmatic Close	Verify programmatically closing a notification causes it to immediately disappear. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Verify notification appears. 4. Click "closeNotification" button. 5. Verify notification is immediately dismissed. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Title Characters & Numbers	Verify title can contain special characters like % or /. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Title Text"to a a long string which contains special characters like %. 3. Click "pop Notification" button. 4. Verify the title text is accurately displayed. 5. teardown() = Click on lock icon in the address bar. Set Show Notifications permission to "Always Ask".
Swipe to Close	Verify notification can be manually dismissed by swiping it (OS X only). 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Click and drag notification to dismiss it. 4. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Require Interaction = True	Verify setting `requireInteraction` to true causes the notification to remain open until explicitly closed (OS X only). 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Set "requireInteraction" dropdown to `true`. 4. Click "pop Notification" button. 5. Verify that the notification persists until explicitly closed. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Require Interaction = False	Verify setting `requireInteraction` to false (default) causes the notification to auto-close after X seconds. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Set "requireInteraction" dropdown to `false`. 4. Click "pop Notification" button. 5. Verify the notification auto dismisses after X seconds. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Redirection = New Tab	Description: Verify clicking on a notification opens a specified URL. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Click on the notification. 4. Verify that https://www.mozilla.org/en-US/ opens in a new tab. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Redirection = Current Tab	Verify setting target opens specified URL in the specified window. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "target" to `_top`. 3. Click "pop Notification" button. 4. Select "Always Show Notifications" from the notifications doorhanger. 5. Verify that https://www.mozilla.org/en-US/ opens in the current tab. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Redirection = Same Window New Tab	Verify setting a target causes a notification to open in the same window. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "target" to `foobar`. 3. Click "pop Notification" button. 4. Select "Always Show Notifications" from the notifications doorhanger. 5. Verify that https://www.mozilla.org/en-US/ opens in a new tab. 6. Click "pop Notification" button. 7. Verify that https://www.mozilla.org/en-US/ opens in the same tab as step 5. 8. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Target = _blank	Verify setting a _blank target causes a notification to open in the new window. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "target" to `_blank` (default). 3. Click "pop Notification" button. 4. Select "Always Show Notifications" from the notifications doorhanger. 5. Verify that https://www.mozilla.org/en-US/ opens in a new tab. 6. Click "pop Notification" button. 7. Verify that https://www.mozilla.org/en-US/ opens in a new tab. 8. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Multi/Repeat Message (Stacking)	Verify clicking the pop Notification button causes the notification to display multiple times. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Click "pop Notification" button 10 times. 4. Verify the notification only appears once (and says "9 new notifications"). 5. Verify there are 11 notifications in the OSX notifications tray. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Hover Over Notification	Verify the notification is not dismissed when hovering over the notification. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Hover over the notification and make sure the notification is not auto-dismissed. 4. Move mouse off of notification. 5. Verify that the notification disappears after X seconds. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Hover Over Submenu	Verify the notification is not dismissed when hovering over an item in the submenu (OS X only). 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Hover over the notification and click the "..." button in the lower-right corner. 4. Click "Disable notifications from this site" menu option. 5. Click "pop Notification" button. 6. Verify notification doorhanger appears. 7. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Preferences - Choose	Verify allowing notifications reports correctly in about:preferences#content 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Open about:preferences#content 4. Click "Choose" button. 5. Verify that https://pdehaan.github.io/ appears in site list and it's status is set to "Allow". 6.teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification Settings: Preferences - Blocking	Verify clicking Notification Settings in the notification causes the preferences page to appear. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Hover over the notification and click the "..." button in the lower-right corner. 4. Click "Notification settings" menu option. 5. Verify that about:preferences#content opens in a new tab. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Preferences - Blocking	Verify blocking notifications reports correctly in about:preferences#content 1. setup()) = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Block Notifications" from the notifications doorhanger. 3. Open about:preferences#content 4. Click "Choose" button. 5. Verify that https://pdehaan.github.io/ appears in site list and it's status is set to "Block". 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Title > Size and Text Wrapping	Verify a long "Title" doesn't cause the notification to re-size. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Title"to a a long string. 3. Click "pop Notification" button. 4. Verify the title text displays and doesn't re-size notification popup. Width remains static, length may adjust. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Body > Size and Text Wrapping	Verify a long "Body Text" doesn't cause the notification to re-size. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Body Text"to a a long string. 3. Click "pop Notification" button. 4. Verify the body text displays text and doesn't re-size notification popup. Width remains static, length may adjust. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Title Characters & Numbers	Verify title can contain special characters like % or /. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Title Text"to a a long string which contains special characters like %. 3. Click "pop Notification" button. 4. Verify the title text is accurately displayed. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Body Text Characters & Numbers	Verify body can contain special characters like % or /. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Body Text"to a a long string which contains special characters like % or an emoji like :D 3. Click "pop Notification" button. 4. Verify the body text is accurately displayed. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Pop-up Settings > Do not disturb me until I restart Nightly	1. Access https://pdehaan.github.io/push-notification-test/ 2. Click the pop Notification button 3. Set the hanger to Always Receive Notifications 4. Verify a popup notification is displayed 5. Click the settings icon on the notification pop-up 6. Select the option Do not disturb me until I restart Nightly 7. Click pop Notification button several times and ensure the pop-up dialog for the notification does not appear 8. Click the lock icon and select Always Ask 9. Close and restart the browser
Content > Block pop-up windows (about:preferences#content)	1. Access https://pdehaan.github.io/push-notification-test/ 2. Click the pop Notification button 3. Set the hanger to Always Receive Notifications 4. Verify a popup notification is displayed 5. Close the popup 6. Select Tools > Options and then click Content on left side bar 7. Check the box Block pop-up windows 8. Return to https://pdehaan.github.io/push-notification-test/ 9. Click the pop Notification button 10. Verify pop up dialog for the notification does not display 11. Click on lock icon in the address bar and set Show Notifications permission to "Always Ask".
Pop-up Settings > Disable notification from pdehaan.github.io	1. Access https://pdehaan.github.io/push-notification-test/ 2. Click the pop Notification button 3. Set the hanger to Always Receive Notifications 4. Verify a popup notification is displayed 5. Click the settings icon on the notification pop-up 6. Select the option Disable notification from pdehaan.github.io 7. Verify the pop-up for the Notification dialog is removed 8. Click the pop Notification button 9. Verify the door hanger is displayed with default set to Always Receive Notification
Pop-up Settings > Notification settings	1. Access https://pdehaan.github.io/push-notification-test/ 2. Click the lock icon and set the Permission option to Always Ask 3. Click the pop Notification button 4. Verify the door hanger displays 5. Set the option to Always Receive Notifications 6. Verify the pop-up dialog for the notification is displayed 7. Click the Settings icon on the pop-up 8. Select the option Notification Settings 9. Verify the current window opens a new tab for Options (about:preferences#content) 10. Click the lock icon and set the option to Always Ask

Sign off

Criteria All the test cases must be executed. All the blocker, critical, major bugs have been fixed.

PUSH API

Common

Shared steps between Push API tests:

setup()

1. Open https://pdehaan.github.io/push-notification-test/
2. Click "register Service Worker" button.

teardown()

1. Click "Unregister Service Worker" button.

Tests

Name: TBD
Description: Verify registering a service worker causes it to appear in about:serviceworkers.
Steps:

1. setup()
2. Verify HTML output is:

   registering service worker

   registered service worker. scope: https://pdehaan.github.io/push-notification-test/

3. Click "Check Service Worker" button.
4. Verify HTML output is:

   checking service worker

   Service worker active

5. Open about:serviceworkers
6. Verify service worker is registered to https://pdehaan.github.io/ origin.
7. Go back to https://pdehaan.github.io/push-notification-test/
8. Click "Unregister Service Worker" button.
9. Open about:serviceworkers
10. Verify "No Service Workers registered." message (refresh page if it was previously open tab).
11. teardown()

Name: TBD
Description: Verify sending a message to a service worker causes it to echo the message.
Steps:

1. setup()
2. Click "subscribe to push" button.
3. Click "sendMsgToSW" button.
4. Verify "messageChannel.port1.onmessage: SW echo: HelloWorld" output on page.
5. Change text box value to "999".
6. Click "sendMsgToSW" button.
7. Verify "messageChannel.port1.onmessage: SW echo: 999" output on page.
8. teardown()

TODO
Test's for:

• Quota, the limit of silent push messages that a user agent (UA) can receive when off web site before UA is unsubscribed from push for that domain. Approximately 30 messages if send every five minutes when off site, less for longer duration.
• Restart browser and off site behaviors
• TTL - time which a message will be dropped if UA is is not online. For example: if TTL=60 and a UA is offline for 59 seconds, they should receive the push message when they come online. If online after 61 seconds, message will be dropped, and UA will not receive message.
• Test different browser modes: e10s/Multi-process mode, Private browsing mode (ServiceWorkers don't work in that mode)
• Test unregistering service worker and verify notifications were not sent.
• Test disable push events should block any push message communication to device.
• Test Do Not Disturb till Fx is restarted.
• Test edge case - multiple browser profiles and multiple tabs doing SW/push subscription.

PUSH NOTIFICATIONS - FENNEC

Test workbook - click here

PUSH API - REFERENCE

Push WebSocket Endpoints:

source:

https://mana.mozilla.org/wiki/display/SVCOPS/SimplePush#SimplePush-Deployments

PUSH ENDPOINTS - STAGE:

Production:	wss://push.services.mozilla.com/
Production:	wss://updates.push.services.mozilla.com/
Staging:	wss://autopush.stage.mozaws.net/
Staging:	wss://updates-autopush.stage.mozaws.net/
Dev:	wss://autopush.dev.mozaws.net/

STATUS / HEALTH:

Production:	https://updates.push.services.mozilla.com/health
Production:	https://updates.push.services.mozilla.com/status
Staging:	https://updates-autopush.stage.mozaws.net/health

CONTACTS

• pdehaan@mozilla
• rpappalardo@mozilla
• ryanvm@mozilla