QA/Push Notifications

From MozillaWiki
< QA
Revision as of 23:54, 29 October 2015 by Mfunches-sv (talk | contribs) (→‎Tests)
Jump to navigation Jump to search

Test Plan (Overview) hot.gif

SUMMARY

  • The purpose of this wiki is to serve as a general testplan for verifying desktop Push notifications.


OWNERSHIP

  • Manual tests will be defined / outlined and executed by the Cloud Services QA Team
  • Manual testing will be handed off to Firefox Quality Engineering Team
  • Automated E2E smoke tests will be created by the Cloud Services QA Team
  • Any further feature automation is TBD (by Firefox Quality Engineering Team)


TEST RESULTS

  • Test results will be posted in Bugzilla (component: TBD).


SIGNOFF CRITERIA Cloud Services QA team will verify that:

  • Desktop push notifications are working properly in FF Nightly.
  • Interaction between push-service (autopush[3]), ServiceWorkers[1], and desktop UI is working as expected.


Testplan (Overview)

REFERENCE


Test Cases

NOTIFICATIONS API

Common

Shared steps between Notifications tests:

setup()

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Click pop Notification button.

teardown()

  1. Click on lock icon in the address bar.
  2. Set Show Notifications permission to "Always Ask".

Tests

Notes: Manual/Automation: Test will be conducted manually Browser Window: Test will be conducted in standard browser window and Private browser windows Platforms: 32bit/64 bit Windows 7, Vista, 8, 8.1, 10; Ubuntu 14.04 (x64) and Mac OS X 10.9.5.

Name: Doorhanger default state test
Description: Verify the doorhanger reports if the site has requested any notification permissions.
Steps:

  1. Click the lock icon in the address bar.
  2. Verify the permissisons section says "You have not granted this site any special permissions."


Name: Notification: Pop-up Dialog
Description: Verify that trying to display a notification causes a dialog to appear.
Steps:

  1. setup()
  2. Click the "close" button in the upper right corner of the notifications doorhanger.
  3. Click "pop Notification" button.
  4. Verify the notification doorhanger appears again and requests permissions.


Name: Notification: Close Dialog by External Means / Non-Close Button
Description: Verify clicking outside of a notification dismisses it automatically (OS X only).
Steps:

  1. setup()
  2. Click outside of the notifications doorhanger.
  3. Verify the notifications doorhanger is dismissed.


Name: Notification: Learn More Link
Description: Verify clicking on the "Learn more" link opens the target page.
Steps:

  1. setup()
  2. Click the "Learn more..." link in the notifications doorhanger.
  3. Verify the browser redirects to https://www.mozilla.org/en-US/firefox/push/ (in a new tab).

Name: Notification: Auto Close (Timeout)
Description: Verify that a notification is auto-closed after X seconds.
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Verify the notification appears and auto-dismisses after X seconds.
  4. teardown()


Name: Notification: Blocking Notifications
Description: Verify that blocking notifications prevents notifications from displaying.
Steps:

  1. setup()
  2. Select "Always Block Notifications" from the notifications doorhanger.
  3. Verify no notification appears.
  4. Click "pop Notification" button.
  5. Verify no notification appears.
  6. teardown()


Name: Notification: Not Now
Description: Verify clicking "Not Now" from the notification permissions doorhanger causes the permissions popup to appear again.
Steps:

  1. setup()
  2. Select "Not Now" from the notifications doorhanger.
  3. Verify no notification appears.
  4. Click "pop Notification" button.
  5. Verify notification doorhanger appears again.
  6. teardown()


Name: Notification: Permission Changes
Description: Verify changing notification permissions in the address bar causes the notification permissions screen to reappear.
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Verify notification appears and auto-dismisses after X seconds.
  4. Click the lock button in the address bar.
  5. Change notification permissions to "Always Ask".
  6. Click "pop Notification" button.
  7. Verify notification doorhanger appears again.
  8. Select "Always Show Notifications" from the notifications doorhanger.
  9. Verify notification appears.
  10. Click the lock button in the address bar.
  11. Change notification permissions to "Block".
  12. Click "pop Notification" button.
  13. Verify no notification appears.
  14. Click the lock button in the address bar.
  15. Change notification permissions to "Allow".
  16. Click "pop Notification" button appears.
  17. Verify notification appears.
  18. teardown()


Name: Notification: Programmatic Close
Description: Verify programmatically closing a notification causes it to immediately disappear.
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Verify notification appears.
  4. Click "closeNotification" button.
  5. Verify notification is immediately dismissed.
  6. teardown()


Name: Notification: Title Characters & Numbers
Description: Verify title can contain special characters like % or /.
Steps:

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Set "Title Text"to a a long string which contains special characters like %.
  3. Click "pop Notification" button.
  4. Verify the title text is accurately displayed.
  5. teardown()

Name: Notification: Swipe to Close
Description: Verify notification can be manually dismissed by swiping it (OS X only).
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Click and drag notification to dismiss it.
  4. teardown()


Name: Push: Register Service Worker
Description: Verify registering a service worker causes it to appear in about:serviceworkers..
Steps:

  1. setup()
  2. Verify HTML output is:
    registering service worker
    registered service worker. scope: https://pdehaan.github.io/push-notification-test/
  1. Click "Check Service Worker" button.
  2. Verify HTML output is:
    checking service worker
    Service worker active
  2. teardown()


Name: Notification: Require Interaction = True
Description: Verify setting `requireInteraction` to true causes the notification to remain open until explicitly closed (OS X only).
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Set "requireInteraction" dropdown to `true`.
  4. Click "pop Notification" button.
  5. Verify that the notification persists until explicitly closed.
  6. teardown()


Name: Notification: Require Interaction = False
Description: Verify setting `requireInteraction` to false (default) causes the notification to auto-close after X seconds.
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Set "requireInteraction" dropdown to `false`.
  4. Click "pop Notification" button.
  5. Verify the notification auto dismisses after X seconds.
  6. teardown()


Name: Notification: Redirection = New Tab
Description: Verify clicking on a notification opens a specified URL.
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Click on the notification.
  4. Verify that https://www.mozilla.org/en-US/ opens in a new tab.
  5. teardown()


Name: Notification: Redirection = Current Tab
Description: Verify setting target opens specified URL in the specified window.
Steps:

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Set "target" to `_top`.
  3. Click "pop Notification" button.
  4. Select "Always Show Notifications" from the notifications doorhanger.
  5. Verify that https://www.mozilla.org/en-US/ opens in the current tab.
  6. teardown()


Name: Notification: Redirection = Same Window New Tab
Description: Verify setting a target causes a notification to open in the same window.
Steps:

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Set "target" to `foobar`.
  3. Click "pop Notification" button.
  4. Select "Always Show Notifications" from the notifications doorhanger.
  5. Verify that https://www.mozilla.org/en-US/ opens in a new tab.
  6. Click "pop Notification" button.
  7. Verify that https://www.mozilla.org/en-US/ opens in the same tab as step 5.
  8. teardown()


Name: Notification: Target = _blank
Description: Verify setting a _blank target causes a notification to open in the new window.
Steps:

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Set "target" to `_blank` (default).
  3. Click "pop Notification" button.
  4. Select "Always Show Notifications" from the notifications doorhanger.
  5. Verify that https://www.mozilla.org/en-US/ opens in a new tab.
  6. Click "pop Notification" button.
  7. Verify that https://www.mozilla.org/en-US/ opens in a new tab.
  8. teardown()


Name: Notification: Multi/Repeat Message (Stacking)
Description: Verify clicking the pop Notification button causes the notification to display multiple times.
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Click "pop Notification" button 10 times.
  4. Verify the notification only appears once (and says "9 new notifications").
  5. Verify there are 11 notifications in the OSX notifications tray.
  6. teardown()


Name: Notification: Hover Over Notification
Description: Verify the notification is not dismissed when hovering over the notification.
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Hover over the notification and make sure the notification is not auto-dismissed.
  4. Move mouse off of notification.
  5. Verify that the notification disappears after X seconds.
  6. teardown()


Name: Notification: Hover Over Submenu
Description: Verify the notification is not dismissed when hovering over an item in the submenu (OS X only).
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Hover over the notification and click the "..." button in the lower-right corner.
  4. Click "Disable notifications from this site" menu option.
  5. Click "pop Notification" button.
  6. Verify notification doorhanger appears.
  7. teardown()


Name: Notification Settings: Preferences - Blocking
Description: Verify clicking Notification Settings in the notification causes the preferences page to appear.
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Hover over the notification and click the "..." button in the lower-right corner.
  4. Click "Notification settings" menu option.
  5. Verify that about:preferences#content opens in a new tab.
  6. teardown()


Name: Notification: Preferences - Choose
Description: Verify allowing notifications reports correctly in about:preferences#content
Steps:

  1. setup()
  2. Select "Always Show Notifications" from the notifications doorhanger.
  3. Open about:preferences#content
  4. Click "Choose" button.
  5. Verify that https://pdehaan.github.io/ appears in site list and it's status is set to "Allow".
  6. teardown()


Name: Notification: Preferences - Blocking
Description: Verify blocking notifications reports correctly in about:preferences#content
Steps:

  1. setup()
  2. Select "Always Block Notifications" from the notifications doorhanger.
  3. Open about:preferences#content
  4. Click "Choose" button.
  5. Verify that https://pdehaan.github.io/ appears in site list and it's status is set to "Block".
  6. teardown()


Name: Notification: Title > Size and Text Wrapping
Description: Verify a long "Title" doesn't cause the notification to resize.
Steps:

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Set "Title"to a a long string.
  3. Click "pop Notification" button.
  4. Verify the title text truncates and doesn't re-size notification popup. Width remains static, length may adjust.
  5. teardown()


Name: Notification: Body > Size ant Text Wrapping
Description: Verify a long "Body Text" doesn't cause the notification to resize.
Steps:

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Set "Body Text"to a a long string.
  3. Click "pop Notification" button.
  4. Verify the body text truncates and doesn't re-size notification popup. Width remains static, length may adjust.
  5. teardown()


Name: Notification: Title Characters & Numbers
Description: Verify title can contain special characters like % or /.
Steps:

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Set "Title Text"to a a long string which contains special characters like %.
  3. Click "pop Notification" button.
  4. Verify the title text is accurately displayed.
  5. teardown()


Name: Notification: Body Text Characters & Numbers
Description: Verify body can contain special characters like % or /.
Steps:

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Set "Body Text"to a a long string which contains special characters like % or an emoji like :D
  3. Click "pop Notification" button.
  4. Verify the body text is accurately displayed.
  5. teardown()


Name: Push: Register Service Worker
Description: Verify registering a service worker causes it to appear in about:serviceworkers.
Steps:

  1. setup()
  2. Verify HTML output is:
    registering service worker
    registered service worker. scope: https://pdehaan.github.io/push-notification-test/
  1. Click "Check Service Worker" button.
  2. Verify HTML output is:
    checking service worker
    Service worker active
  1. Open about:serviceworkers
  2. Verify service worker is registered to https://pdehaan.github.io/ origin.
  3. Go back to https://pdehaan.github.io/push-notification-test/
  4. Click "Unregister Service Worker" button.
  5. Open about:serviceworkers
  6. Verify "No Service Workers registered." message (refresh page if it was previously open tab).
  7. teardown()


Name: Push: Service Worker
Description: Verify sending a message to a service worker causes it to echo the message.
Steps:

  1. setup()
  2. Click "subscribe to push" button.
  3. Click "sendMsgToSW" button.
  4. Verify "messageChannel.port1.onmessage: SW echo: HelloWorld" output on page.
  5. Change text box value to "999".
  6. Click "sendMsgToSW" button.
  7. Verify "messageChannel.port1.onmessage: SW echo: 999" output on page.
  8. teardown()


Sign off

Criteria All the test cases must be executed. All the blocker, critical, major bugs have been fixed.

PUSH API

Common

Shared steps between Push API tests:

setup()

  1. Open https://pdehaan.github.io/push-notification-test/
  2. Click "register Service Worker" button.

teardown()

  1. Click "Unregister Service Worker" button.

Tests

Name: TBD
Description: Verify registering a service worker causes it to appear in about:serviceworkers.
Steps:

  1. setup()
  2. Verify HTML output is:
    registering service worker
    registered service worker. scope: https://pdehaan.github.io/push-notification-test/
  3. Click "Check Service Worker" button.
  4. Verify HTML output is:
    checking service worker
    Service worker active
  5. Open about:serviceworkers
  6. Verify service worker is registered to https://pdehaan.github.io/ origin.
  7. Go back to https://pdehaan.github.io/push-notification-test/
  8. Click "Unregister Service Worker" button.
  9. Open about:serviceworkers
  10. Verify "No Service Workers registered." message (refresh page if it was previously open tab).
  11. teardown()


Name: TBD
Description: Verify sending a message to a service worker causes it to echo the message.
Steps:

  1. setup()
  2. Click "subscribe to push" button.
  3. Click "sendMsgToSW" button.
  4. Verify "messageChannel.port1.onmessage: SW echo: HelloWorld" output on page.
  5. Change text box value to "999".
  6. Click "sendMsgToSW" button.
  7. Verify "messageChannel.port1.onmessage: SW echo: 999" output on page.
  8. teardown()

TODO
Test's for:

  • Quota, the limit of silent push messages that a user agent (UA) can receive when off web site before UA is unsubscribed from push for that domain. Approximately 30 messages if send every five minutes when off site, less for longer duration.
  • Restart browser and off site behaviors
  • TTL - time which a message will be dropped if UA is is not online. For example: if TTL=60 and a UA is offline for 59 seconds, they should receive the push message when they come online. If online after 61 seconds, message will be dropped, and UA will not receive message.
  • Test different browser modes: e10s/Multi-process mode, Private browsing mode (ServiceWorkers don't work in that mode)
  • Test unregistering service worker and verify notifications were not sent.
  • Test disable push events should block any push message communication to device.
  • Test Do Not Disturb till Fx is restarted.
  • Test edge case - multiple browser profiles and multiple tabs doing SW/push subscription.

PUSH API - REFERENCE

Push WebSocket Endpoints:

source: https://mana.mozilla.org/wiki/display/SVCOPS/SimplePush#SimplePush-Deployments

PUSH ENDPOINTS - STAGE:

Production: wss://push.services.mozilla.com/
Production: wss://updates.push.services.mozilla.com/
Staging: wss://autopush.stage.mozaws.net/
Staging: wss://updates-autopush.stage.mozaws.net/
Dev: wss://autopush.dev.mozaws.net/

STATUS / HEALTH:

Production: https://updates.push.services.mozilla.com/health
Production: https://updates.push.services.mozilla.com/status
Staging: https://updates-autopush.stage.mozaws.net/health


CONTACTS

  • pdehaan@mozilla
  • rpappalardo@mozilla
  • ryanvm@mozilla
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Push_Notifications&oldid=1103354"