B2G/QA/Smoke Tests

From MozillaWiki
< B2G‎ | QA
Revision as of 23:20, 5 February 2015 by Pragmatic (talk | contribs) (→‎Smoke Tests: limited TOC depth)
Jump to navigation Jump to search

High Severity Bugs

  • Any action that bricks the device.
  • Any action that causes the device not to come up again after reboot.
  • Any action that causes the device to crash and reboot.
  • The following Sanity Test Cases fail without a workaround:
    • [RIL] Correct Radio Interface Layer is installed
    • [OTA] Over The Air update is successful
    • [Homescreen] The Homescreen is viewable
    • [Dialer] Can make an Outbound Call twice
    • [Dialer] Can make an Inbound Call twice
    • [SMS] Can send an Outbound Message twice
    • [SMS] Can receive an Inbound Message twice
    • [Camera] Can take a photo
    • [Camera] Can view the photo taken
    • [Browser] Can browse for 1-2 minutes over WiFi

Note: The telephony tests are done twice to make sure that the process cleans up after itself when the first instance is over so that the second instance is also successful.

Google Spreadsheet that tracks daily test logs for Sanity Test Cases.

Escalation Strategy

  • When a High Severity Bug is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
  • A notification email is sent by QAnalysts to the following mailing lists: mozilla-dev-b2g@lists.mozilla.org, dev-b2g@lists.mozilla.org, dev-gaia@lists.mozilla.org and b2g-release-drivers@mozilla.org
  • The IRC channel owners of #b2g and #gaia are notified in their respective channels.
  • A bug is filed requesting OTA to be blocked.
  • A regression window for the root cause of the bug is identified.
  • The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
  • The Whiteboard tag "[backout-needed]" is added to the bug.
  • The module owner is also needinfo?ed.
  • QA tracks the bug until the commit has been backed out. Bugzilla Query for the Whiteboard tag "[backout-needed]": http://mzl.la/1v2bcJJ
  • QA requests for a new nightly build to be created.
  • When a new build becomes available, QA verifies that the bug no longer reproduces.
  • If the bug no longer reproduces, the bug requesting OTA to be reopened and a request is made to enable OTA again.

Smoke Tests by Functional Area

Google Spreadsheet that tracks daily test logs for Smoke Test Cases.

Communications

Dialer

  • 1298 Verify an outbound call can be made and that the Call Volume can be adjusted
  • 6076 Verify that touch tones work using Mozilla's conference line.

Messages

  • 10742 Send an MMS with picture attached

SIM

  • 14361 Dialer Send SIM control codes for pin change
  • 10739 Send MMS to secondary SIM card (Auto retrieve on with roaming)

Telephony/Networking

Bluetooth

  • 6071 Verify that the user can enable Bluetooth and pair to a Bluetooth enabled computer/another device.
  • 6070 Pair and receive an incoming call using bluetooth
  • 9346 Music can be played and stoped remotely by user (DRAFT without UI stuffs)

NFC

WiFi

  • 6075 Verify the user can connect to WPA/PSK & WPA2/AES WiFi access points

Geolocation

Internet Sharing

Media

Camera

  • 2460 Verify the user can take a picture with camera from lockscreen.
  • 14789 Take pictures using Face recognition and Tap to Focus

Video

  • 2478 Play the video you recorded. Check for video and sound

Music

  • 4031 Artist View - Verify the user can play a single song.

Gallery

  • 14645 Test you can view a thumbnail in large mode

Systems Front End

Browser

Homescreen

  • 14644 Battery Charge indicator given when charging
  • 6117 Verify the user can delete a packaged app
  • 1333 Marketplace - Verify the user can launch market installed hosted apps.
  • 5856 Install and launch Facebook and Twitter apps from Homescreen Search
  • 6116 Install and launch a packaged app
  • 6786 Verify a market installed hosted app can be installed and removed.

Graphics

Systems Platform

Lockscreen

  • 15243 Verify Lockscreen Display after timeout

OTA

  • 2313 System Updates - Success over Wifi
  • 14028 Verify Settings Data is Preserved after OTA
  • 14029 Verify Homescreen Data is Preserved after OTA
  • 14030 Verify Marketplace Data is Preserved after OTA
  • 14039 Verify Contact Data is Preserved after OTA
  • 14040 Verify Music Data is Preserved after OTA
  • 14041 Verify FM Radio Data is Preserved after OTA
  • 14042 Verify Notification Data is Preserved after OTA
  • 14043 Verify Usage app Data is Preserved after OTA
  • 14044 Verify Browser Data is Preserved after OTA
  • 14045 Verify Email Data is Preserved after OTA
  • 14051 Verify Gallery Data is Preserved after OTA
  • 14052 Verify Calendar Data is Preserved after OTA
  • 14053 Verify Clock Data is Preserved after OTA
  • 14054 Verify Messaging Data is Preserved after OTA

Multimedia Platform

Productivity

Email

  • 10744 Notification for E-mail will be display

Clock

  • 1772 [Alarm] Test that when the alarm triggers, the phone vibrates and an audible alarm is triggered

Storage

  • 6072 Verify USB Mass Storage Functionality is correct

Metrics

Localization

Marketplace

  • 1332 Marketplace - Verify the user can install market installed hosted app.

Firefox Accounts

  • 13006 Successful Sign up in Settings

Smoke Test Blockers

Bugzilla Query for the Whiteboard tag "smoketest": http://mzl.la/16lkWbM

Escalation Strategy

  • When a Smoke Test Blocker is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
  • The QA Team is notified in the the #fxosqa IRC channel.
  • A regression window for the root cause of the bug is identified.
  • The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
  • The Whiteboard tag "[backout-needed]" is added to the bug.
  • The module owner is also needinfo?ed.
  • A Smoke Test report email is sent daily by the Firefox OS QA team to the following mailing lists: fxosqa-report-watchlist@mozilla.org and b2g-release-drivers@mozilla.org
  • QA tracks the bug until the commit has been backed out.
  • QA requests for a new nightly build to be created.
  • When a new build becomes available, QA verifies that the bug no longer reproduces.

Test Automation

Retrieved from "https://wiki.allizom.org/index.php?title=B2G/QA/Smoke_Tests&oldid=1054098"