B2G/QA/Smoke Tests

High Severity Bugs

• Any action that bricks the device.
• Any action that causes the device not to come up again after reboot.
• Any action that causes the device to crash and reboot.
• The following Sanity Test Cases fail without a workaround:
  • [RIL] Correct Radio Interface Layer is installed
  • [OTA] Over The Air update is successful
  • [Homescreen] The Homescreen is viewable
  • [Dialer] Can make an Outbound Call twice
  • [Dialer] Can make an Inbound Call twice
  • [SMS] Can send an Outbound Message twice
  • [SMS] Can receive an Inbound Message twice
  • [Camera] Can take a photo
  • [Camera] Can view the photo taken
  • [Browser] Can browse for 1-2 minutes over WiFi

Note: The telephony tests are done twice to make sure that the process cleans up after itself when the first instance is over so that the second instance is also successful.

Google Spreadsheet that tracks daily test logs for Sanity Test Cases.

Escalation Strategy

• When a High Severity Bug is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
• A notification email is sent by QAnalysts to the following mailing lists: mozilla-dev-b2g@lists.mozilla.org, dev-b2g@lists.mozilla.org, dev-gaia@lists.mozilla.org and b2g-release-drivers@mozilla.org
• The IRC channel owners of #b2g and #gaia are notified in their respective channels.
• A bug is filed requesting OTA to be blocked.
• A regression window for the root cause of the bug is identified.
• The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
• The Whiteboard tag "[backout-needed]" is added to the bug.
• The module owner is also needinfo?ed.
• QA tracks the bug until the commit has been backed out. Bugzilla Query for the Whiteboard tag "[backout-needed]": http://mzl.la/1v2bcJJ
• QA requests for a new nightly build to be created.
• When a new build becomes available, QA verifies that the bug no longer reproduces.
• If the bug no longer reproduces, the bug requesting OTA to be blocked is updated.

Smoke Tests by Functional Area

Google Spreadsheet that tracks daily test logs for Smoke Test Cases.

Communications

Dialer

1298 [DIALER] Verify an outbound call can be made and that the Call Volume can be adjusted 6076 [Dialer] Verify that touch tones work using Mozilla's conference line.

Messages

10742 [MMS] Send an MMS with picture attached

SIM

14361 [Comms]Dialer Send SIM control codes for pin change 10739 [DSDS] Send MMS to secondary SIM card (Auto retrieve on with roaming)

Telephony/Networking

Bluetooth

6071 [Bluetooth] Verify that the user can enable Bluetooth and pair to a Bluetooth enabled computer/another device. 6070 [Bluetooth] Pair and receive an incoming call using bluetooth 9346 [Bluetooth] Music can be played and stoped remotely by user (DRAFT without UI stuffs)

WiFi

6075 [WiFi] Verify the user can connect to WPA/PSK & WPA2/AES WiFi access points

Media

2460 [CAMERA] Verify the user can take a picture with camera from lockscreen. 14789 [Camera] Take pictures using Face recognition and Tap to Focus 2478 [Video] Play the video you recorded. Check for video and sound 4031 [Music] Artist View - Verify the user can play a single song. 14645 [Gallery] Test you can view a thumbnail in large mode

Systems Front End

6072 [Settings] Verify USB Mass Storage Functionality is correct 6073 [Browser] Verify that videos can be streamed and display correctly on YouTube.com 15267 [Browser] Test Zooming on website 14644 [Homescreen] Battery Charge indicator given when charging 6117 [HomeScreen] Verify the user can delete a packaged app 1333 [HOMESCREEN] Marketplace - Verify the user can launch market installed hosted apps. 5856 [Home Screen] Install and launch Facebook and Twitter apps from Homescreen Search 6116 [Homescreen] Install and launch a packaged app 6786 [APP REMOVAL] Verify a market installed hosted app can be installed and removed.

Systems Platform

2313 [Updates] System Updates - Success over Wifi 15243 [Lockscreen] Verify Lockscreen Display after timeout 14028 Verify Settings Data is Preserved after OTA 14029 Verify Homescreen Data is Preserved after OTA 14030 Verify Marketplace Data is Preserved after OTA 14039 Verify Contact Data is Preserved after OTA 14040 Verify Music Data is Preserved after OTA 14041 Verify FM Radio Data is Preserved after OTA 14042 Verify Notification Data is Preserved after OTA 14043 Verify Usage app Data is Preserved after OTA 14044 Verify Browser Data is Preserved after OTA 14045 Verify Email Data is Preserved after OTA 14051 Verify Gallery Data is Preserved after OTA 14052 Verify Calendar Data is Preserved after OTA 14053 Verify Clock Data is Preserved after OTA 14054 Verify Messaging Data is Preserved after OTA

Productivity

10744 [EMAIL] Notification for E-mail will be display 1772 [Clock][Alarm] Test that when the alarm triggers, the phone vibrates and an audible alarm is triggered

Marketplace

1332 [MARKETPLACE] Marketplace - Verify the user can install market installed hosted app.

Firefox Accounts

13006 Smoketest: Successful Sign up in Settings

Smoke Test Blockers

Bugzilla Query for the Whiteboard tag "smoketest": http://mzl.la/16lkWbM

Escalation Strategy

• When a Smoke Test Blocker is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
• The QA Team is notified in the the #fxosqa IRC channel.
• A regression window for the root cause of the bug is identified.
• The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
• The Whiteboard tag "[backout-needed]" is added to the bug.
• The module owner is also needinfo?ed.
• A Smoke Test report email is sent daily by the Firefox OS QA team to the following mailing lists: fxosqa-report-watchlist@mozilla.org and b2g-release-drivers@mozilla.org
• QA tracks the bug until the commit has been backed out.
• QA requests for a new nightly build to be created.
• When a new build becomes available, QA verifies that the bug no longer reproduces.

Test Automation

• Gaia UI Tests: https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests