QA/Push Notifications

Test Plan (Overview)

SUMMARY

• The purpose of this wiki is to serve as a general testplan for verifying desktop Push notifications.

OWNERSHIP

• Manual tests will be defined / outlined and executed by the Cloud Services QA Team
• Manual testing will be handed off to sOFTVISION qa tEAM
• Automated E2E smoke tests will be created by the Cloud Services QA Team
• Any further feature automation is TBD (by Firefox Quality Engineering Team)

TEST RESULTS

• Test results will be posted in Bugzilla (component: TBD).

SIGNOFF CRITERIA Cloud Services QA team will verify that:

• Desktop push notifications are working properly in FF Nightly.
• Interaction between push-service (autopush[3]), ServiceWorkers[1], and desktop UI is working as expected.

Testplan (Overview) Platform(s) Windows 10, Windows 8, Windows 8.1, Windows Vista, Windows 7, Windows XP, Ubuntu 14.04, Mac OS 10

Test Name	Description
Notifications Dialog	Verify that trying to display a notification causes a dialog to appear.
Notifications Dialog	Verify that trying to display a notification causes a dialog to appear.
Notification: Learn More	Verify clicking on the "Learn more" link opens the target page.
Notification: Timeout	Verify that a notification is auto-closed after X seconds.
Notification: Blocking	Verify that blocking notifications prevents notifications from displaying.
Notification: Not Now	Verify clicking "Not Now" from the notification permissions doorhanger causes the permissions popup to appear again.
Notification: Permission Change	Verify changing notification permissions in the address bar causes the notification permissions screen to reappear.
Programmatic Close	Verify programmatically closing a notification causes it to immediately disappear.
Notification: Close Button	Verify notification can be dismissed by clicking the close button.
Notification: Require Interaction = True	Verify setting `requireInteraction` to true causes the notification to remain open until explicitly closed (OS X only).
Notification: Redirection	Verify clicking on a notification opens a specified URL.
Notification: Pop Up Multi Message	Verify clicking the pop Notification button causes the notification to display multiple times.
Notification: Hovering Over Notification	Verify the notification is not dismissed when hovering over the notification.
Notification: Hovering Over Submenu	Verify clicking "Do not disturb me until I restart Nightly" option stops the notifications untill browser is restarted.
Notification: Settings & Preferences	Verify clicking Notification Settings in the notification causes the preferences page to appear.
Notification: Preferences Content	Verify allowing notifications reports correctly in about:preferences#content
Notification: Blocking > Preferences Content	Verify blocking notifications reports correctly in about:preferences#content
Push: Register Service Worker	Verify registering a service worker causes it to appear in about:serviceworkers.
Push: Send Message > Service Worker	Verify sending a message to a service worker causes it to echo the message.

REFERENCE

• [1] http://www.w3.org/TR/service-workers/
• [2] http://www.w3.org/TR/push-api/
• [3] autopush: https://github.com/mozilla-services/autopush/
• [4] push-service: https://github.com/mozilla-services/push-service

Test Cases

NOTIFICATIONS API

Common

Shared steps between Notifications tests:

setup()

1. Open https://pdehaan.github.io/push-notification-test/
2. Click pop Notification button.

teardown()

1. Click on lock icon in the address bar.
2. Set Show Notifications permission to "Always Ask".

Friday: In-Depth & Exploratory Tests

Notes: Manual/Automation: Test will be conducted manually Browser Window: Test will be conducted in standard browser window and Private browser windows Platforms: 32bit/64 bit Windows 7, Vista, 8, 8.1, 10; Ubuntu 14.04 (x64) and Mac OS X 10.9.5.

Title	Description	Win 10	Win 8	Win 8.1	Vista	Win 7	Mac OS	Ubuntu
Doorhanger default state test	Verify the doorhanger reports if the site has requested any notification permissions. 1. Click the lock icon in the address bar. 2. Verify the Verify the doorhanger reports if the site has requested any notification permissions.
Pop-up Dialog	Verify that trying to display a notification causes a dialog to appear. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Click the "close" button in the upper right corner of the notifications doorhanger. 3. Click "pop Notification" button. 4. Verify the notification doorhanger appears again and requests permissions.
Close Dialog by External Means / Non-Close Button	Verify clicking outside of a notification dismisses it automatically (OS X only). 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Click outside of the notifications doorhanger. 3. Verify the notifications doorhanger is dismissed.
Learn More Link	Verify clicking on the "Learn more" link opens the target page. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Click the "Learn more..." link in the notifications doorhanger. 3. Verify the browser redirects to https://www.mozilla.org/en-US/firefox/push/ (in a new tab).
Auto Close (Timeout)	Verify that a notification is auto-closed after X seconds. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Verify the notification appears and auto-dismisses after X seconds. 4. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Blocking Notifications	Verify that blocking notifications prevents notifications from displaying. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Block Notifications" from the notifications doorhanger. 3. Verify no notification appears. 4. Click "pop Notification" button. 5. Verify no notification appears. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Not Now	Verify clicking "Not Now" from the notification permissions doorhanger causes the permissions popup to appear again. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Not Now" from the notifications doorhanger. 3. Verify no notification appears. 4. Click "pop Notification" button. 5. Verify notification doorhanger appears again. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Permission Changes	Verify changing notification permissions in the address bar causes the notification permissions screen to reappear. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Verify notification appears and auto-dismisses after X seconds. 4. Click the lock button in the address bar. 5. Change notification permissions to "Always Ask". 6. Click "pop Notification" button. 7. Verify notification doorhanger appears again. 8. Select "Always Show Notifications" from the notifications doorhanger. 9. Verify notification appears. 10. Click the lock button in the address bar. 11. Change notification permissions to "Block". 12. Click "pop Notification" button. 13. Verify no notification appears. 14. Click the lock button in the address bar. 15. Change notification permissions to "Allow". 16. Click "pop Notification" button appears. 17. Verify notification appears. 18. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Programmatic Close	Verify programmatically closing a notification causes it to immediately disappear. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Verify notification appears. 4. Click "closeNotification" button. 5. Verify notification is immediately dismissed. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Title Characters & Numbers	Verify title can contain special characters like % or /. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Title Text"to a a long string which contains special characters like %. 3. Click "pop Notification" button. 4. Verify the title text is accurately displayed. 5. teardown() = Click on lock icon in the address bar. Set Show Notifications permission to "Always Ask".
Swipe to Close	Verify notification can be manually dismissed by swiping it (OS X only). 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Click and drag notification to dismiss it. 4. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Require Interaction = True	Verify setting `requireInteraction` to true causes the notification to remain open until explicitly closed (OS X only). 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Set "requireInteraction" dropdown to `true`. 4. Click "pop Notification" button. 5. Verify that the notification persists until explicitly closed. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Require Interaction = False	Verify setting `requireInteraction` to false (default) causes the notification to auto-close after X seconds. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Set "requireInteraction" dropdown to `false`. 4. Click "pop Notification" button. 5. Verify the notification auto dismisses after X seconds. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Redirection = New Tab	Description: Verify clicking on a notification opens a specified URL. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Click on the notification. 4. Verify that https://www.mozilla.org/en-US/ opens in a new tab. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Redirection = Current Tab	Verify setting target opens specified URL in the specified window. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "target" to `_top`. 3. Click "pop Notification" button. 4. Select "Always Show Notifications" from the notifications doorhanger. 5. Verify that https://www.mozilla.org/en-US/ opens in the current tab. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Redirection = Same Window New Tab	Verify setting a target causes a notification to open in the same window. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "target" to `foobar`. 3. Click "pop Notification" button. 4. Select "Always Show Notifications" from the notifications doorhanger. 5. Verify that https://www.mozilla.org/en-US/ opens in a new tab. 6. Click "pop Notification" button. 7. Verify that https://www.mozilla.org/en-US/ opens in the same tab as step 5. 8. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Target = _blank	Verify setting a _blank target causes a notification to open in the new window. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "target" to `_blank` (default). 3. Click "pop Notification" button. 4. Select "Always Show Notifications" from the notifications doorhanger. 5. Verify that https://www.mozilla.org/en-US/ opens in a new tab. 6. Click "pop Notification" button. 7. Verify that https://www.mozilla.org/en-US/ opens in a new tab. 8. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Multi/Repeat Message (Stacking)	Verify clicking the pop Notification button causes the notification to display multiple times. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Click "pop Notification" button 10 times. 4. Verify the notification only appears once (and says "9 new notifications"). 5. Verify there are 11 notifications in the OSX notifications tray. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Hover Over Notification	Verify the notification is not dismissed when hovering over the notification. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Hover over the notification and make sure the notification is not auto-dismissed. 4. Move mouse off of notification. 5. Verify that the notification disappears after X seconds. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Hover Over Submenu	Verify the notification is not dismissed when hovering over an item in the submenu (OS X only). 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Hover over the notification and click the "..." button in the lower-right corner. 4. Click "Disable notifications from this site" menu option. 5. Click "pop Notification" button. 6. Verify notification doorhanger appears. 7. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Preferences - Choose	Verify allowing notifications reports correctly in about:preferences#content 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Open about:preferences#content 4. Click "Choose" button. 5. Verify that https://pdehaan.github.io/ appears in site list and it's status is set to "Allow". 6.teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification Settings: Preferences - Blocking	Verify clicking Notification Settings in the notification causes the preferences page to appear. 1. setup() = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Show Notifications" from the notifications doorhanger. 3. Hover over the notification and click the "..." button in the lower-right corner. 4. Click "Notification settings" menu option. 5. Verify that about:preferences#content opens in a new tab. 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Preferences - Blocking	Verify blocking notifications reports correctly in about:preferences#content 1. setup()) = Open https://pdehaan.github.io/push-notification-test/ and then Click pop Notification button. 2. Select "Always Block Notifications" from the notifications doorhanger. 3. Open about:preferences#content 4. Click "Choose" button. 5. Verify that https://pdehaan.github.io/ appears in site list and it's status is set to "Block". 6. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Title > Size and Text Wrapping	Verify a long "Title" doesn't cause the notification to re-size. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Title"to a a long string. 3. Click "pop Notification" button. 4. Verify the title text displays and doesn't re-size notification popup. Width remains static, length may adjust. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Body > Size and Text Wrapping	Verify a long "Body Text" doesn't cause the notification to re-size. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Body Text"to a a long string. 3. Click "pop Notification" button. 4. Verify the body text displays text and doesn't re-size notification popup. Width remains static, length may adjust. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Title Characters & Numbers	Verify title can contain special characters like % or /. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Title Text"to a a long string which contains special characters like %. 3. Click "pop Notification" button. 4. Verify the title text is accurately displayed. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".
Notification: Body Text Characters & Numbers	Verify body can contain special characters like % or /. 1. Open https://pdehaan.github.io/push-notification-test/ 2. Set "Body Text"to a a long string which contains special characters like % or an emoji like :D 3. Click "pop Notification" button. 4. Verify the body text is accurately displayed. 5. teardown() = Click on lock icon in the address bar & Set Show Notifications permission to "Always Ask".

Sign off

Criteria All the test cases must be executed. All the blocker, critical, major bugs have been fixed.

PUSH API

Common

Shared steps between Push API tests:

setup()

1. Open https://pdehaan.github.io/push-notification-test/
2. Click "register Service Worker" button.

teardown()

1. Click "Unregister Service Worker" button.

Tests

Name: TBD
Description: Verify registering a service worker causes it to appear in about:serviceworkers.
Steps:

1. setup()
2. Verify HTML output is:

   registering service worker

   registered service worker. scope: https://pdehaan.github.io/push-notification-test/

3. Click "Check Service Worker" button.
4. Verify HTML output is:

   checking service worker

   Service worker active

5. Open about:serviceworkers
6. Verify service worker is registered to https://pdehaan.github.io/ origin.
7. Go back to https://pdehaan.github.io/push-notification-test/
8. Click "Unregister Service Worker" button.
9. Open about:serviceworkers
10. Verify "No Service Workers registered." message (refresh page if it was previously open tab).
11. teardown()

Name: TBD
Description: Verify sending a message to a service worker causes it to echo the message.
Steps:

1. setup()
2. Click "subscribe to push" button.
3. Click "sendMsgToSW" button.
4. Verify "messageChannel.port1.onmessage: SW echo: HelloWorld" output on page.
5. Change text box value to "999".
6. Click "sendMsgToSW" button.
7. Verify "messageChannel.port1.onmessage: SW echo: 999" output on page.
8. teardown()

TODO
Test's for:

• Quota, the limit of silent push messages that a user agent (UA) can receive when off web site before UA is unsubscribed from push for that domain. Approximately 30 messages if send every five minutes when off site, less for longer duration.
• Restart browser and off site behaviors
• TTL - time which a message will be dropped if UA is is not online. For example: if TTL=60 and a UA is offline for 59 seconds, they should receive the push message when they come online. If online after 61 seconds, message will be dropped, and UA will not receive message.
• Test different browser modes: e10s/Multi-process mode, Private browsing mode (ServiceWorkers don't work in that mode)
• Test unregistering service worker and verify notifications were not sent.
• Test disable push events should block any push message communication to device.
• Test Do Not Disturb till Fx is restarted.
• Test edge case - multiple browser profiles and multiple tabs doing SW/push subscription.

PUSH API - REFERENCE

Push WebSocket Endpoints:

source:

https://mana.mozilla.org/wiki/display/SVCOPS/SimplePush#SimplePush-Deployments

PUSH ENDPOINTS - STAGE:

Production:	wss://push.services.mozilla.com/
Production:	wss://updates.push.services.mozilla.com/
Staging:	wss://autopush.stage.mozaws.net/
Staging:	wss://updates-autopush.stage.mozaws.net/
Dev:	wss://autopush.dev.mozaws.net/

STATUS / HEALTH:

Production:	https://updates.push.services.mozilla.com/health
Production:	https://updates.push.services.mozilla.com/status
Staging:	https://updates-autopush.stage.mozaws.net/health

CONTACTS

• pdehaan@mozilla
• rpappalardo@mozilla
• ryanvm@mozilla