MozillaWiki

B2G/QA/Smoke Tests

< B2G‎ | QA
Revision as of 01:34, 6 February 2015 by Pragmatic (talk | contribs) (→‎Systems Front End: →‎First Time Use: added automated tests)

High Severity Bugs

  • Any action that bricks the device.
  • Any action that causes the device not to come up again after reboot.
  • Any action that causes the device to crash and reboot.
  • The following Sanity Test Cases fail without a workaround:
    • [RIL] Correct Radio Interface Layer is installed
    • [OTA] Over The Air update is successful
    • [Homescreen] The Homescreen is viewable
    • [Dialer] Can make an Outbound Call twice
    • [Dialer] Can make an Inbound Call twice
    • [SMS] Can send an Outbound Message twice
    • [SMS] Can receive an Inbound Message twice
    • [Camera] Can take a photo
    • [Camera] Can view the photo taken
    • [Browser] Can browse for 1-2 minutes over WiFi

Note: The telephony tests are done twice to make sure that the process cleans up after itself when the first instance is over so that the second instance is also successful.

Google Spreadsheet that tracks daily test logs for Sanity Test Cases.

Escalation Strategy

  • When a High Severity Bug is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
  • A notification email is sent by QAnalysts to the following mailing lists: mozilla-dev-b2g@lists.mozilla.org, dev-b2g@lists.mozilla.org, dev-gaia@lists.mozilla.org and b2g-release-drivers@mozilla.org
  • The IRC channel owners of #b2g and #gaia are notified in their respective channels.
  • A bug is filed requesting OTA to be blocked.
  • A regression window for the root cause of the bug is identified.
  • The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
  • The Whiteboard tag "[backout-needed]" is added to the bug.
  • The module owner is also needinfo?ed.
  • QA tracks the bug until the commit has been backed out. Bugzilla Query for the Whiteboard tag "[backout-needed]": http://mzl.la/1v2bcJJ
  • QA requests for a new nightly build to be created.
  • When a new build becomes available, QA verifies that the bug no longer reproduces.
  • If the bug no longer reproduces, the bug requesting OTA to be reopened and a request is made to enable OTA again.

Smoke Tests by Functional Area

Google Spreadsheet that tracks daily test logs for Smoke Test Cases.

Communications

Dialer

  • 1298 Verify an outbound call can be made and that the Call Volume can be adjusted
  • 6076 Verify that touch tones work using Mozilla's conference line.

Messages

  • 10742 Send an MMS with picture attached

SIM

  • 14361 Dialer Send SIM control codes for pin change
  • 10739 Send MMS to secondary SIM card (Auto retrieve on with roaming)

Contacts

Emergency Call

  • 15186 (automated) A regular phone number should not be dialed out on the emergency dialer

Telephony/Networking

Bluetooth

  • 6071 Verify that the user can enable Bluetooth and pair to a Bluetooth enabled computer/another device.
  • 6070 Pair and receive an incoming call using bluetooth
  • 9346 Music can be played and stoped remotely by user (DRAFT without UI stuffs)

NFC

WiFi

  • 6075 Verify the user can connect to WPA/PSK & WPA2/AES WiFi access points

FM Radio

Geolocation

Internet Sharing

Cost Control

  • 8938 (automated) Check if the data alert is triggered after 1-2 minutes of browsing over cellular data
  • (automated) Turn on Data use alert in Cost Control's First Time User wizard and ensure that Mobile Usage is checked
  • (automated) Browse over WiFi and reset WiFi usage. Check that usage stats are zero.

Media

Camera

  • 2460 Verify the user can take a picture with camera from lockscreen.
  • 14789 Take pictures using Face recognition and Tap to Focus
  • 1325 (automated) Launch the Camera app, take a photo, check that the thumbnail is visible and that the photo has been saved on the SD Card
  • 2477 (automated) Launch the Camera app, switch to video mode, record 3 seconds of video, check that the video has been saved on the SD Card
  • 3324 (automated) Record 10 seconds of video, tap preview icon to play back the video, then pause it. Tap the options icon, and delete the file. Check the user is back in camera preview mode, and there is no preview icon anymore. Verify the video is deleted from the SD Card.
  • 1325 (automated) Launch the Camera app and toggle between flash modes "on", "off" and "auto". In each mode, take a photo, check that the thumbnail is visible and that the photo has been saved on the SD Card.
  • 1325 (automated) Launch the Camera app, take a photo, check that the thumbnail is visible and that the photo has been saved on the SD Card. Repeat this action twice more.

Video

  • 2478 Play the video you recorded. Check for video and sound

Music

  • 4031 Artist View - Verify the user can play a single song.

Gallery

  • 14645 Test you can view a thumbnail in large mode

Systems Front End

Browser

Homescreen

  • 14644 Battery Charge indicator given when charging
  • 6117 Verify the user can delete a packaged app
  • 1333 Marketplace - Verify the user can launch market installed hosted apps.
  • 5856 Install and launch Facebook and Twitter apps from Homescreen Search
  • 6116 Install and launch a packaged app
  • 6786 Verify a market installed hosted app can be installed and removed.

Task Manager

  • 1917 (automated) Launch the Clock and Gallery apps. Long press the Home button to get the cards view and close each app. Check that the apps no longer appear in the cards view and the "No recent apps" message should be displayed.
  • 2462 (automated) Launch the Clock and Calendar apps. Long press the Home button to get the cards view and swipe to the previous app. Tap the card view of the app to bring it up again.

First Time Use

Graphics

Systems Platform

Lockscreen

  • 15243 Verify Lockscreen Display after timeout

OTA

  • 2313 System Updates - Success over Wifi
  • 14028 Verify Settings Data is Preserved after OTA
  • 14029 Verify Homescreen Data is Preserved after OTA
  • 14030 Verify Marketplace Data is Preserved after OTA
  • 14039 Verify Contact Data is Preserved after OTA
  • 14040 Verify Music Data is Preserved after OTA
  • 14041 Verify FM Radio Data is Preserved after OTA
  • 14042 Verify Notification Data is Preserved after OTA
  • 14043 Verify Usage app Data is Preserved after OTA
  • 14044 Verify Browser Data is Preserved after OTA
  • 14045 Verify Email Data is Preserved after OTA
  • 14051 Verify Gallery Data is Preserved after OTA
  • 14052 Verify Calendar Data is Preserved after OTA
  • 14053 Verify Clock Data is Preserved after OTA
  • 14054 Verify Messaging Data is Preserved after OTA

Multimedia Platform

Storage

  • 6072 Verify USB Mass Storage Functionality is correct

Productivity

Email

  • 10744 Notification for E-mail will be display

Clock

  • 1772 [Alarm] Test that when the alarm triggers, the phone vibrates and an audible alarm is triggered
  • 1772 1775 (automated) Create a new alarm, give it a label and save it. Check that the confirmation message appears with a countdown. Edit the alarm and set it to trigger 1 minute from the current time. Turn off the alarm and turn it back on. Press the Home button and wait for the alarm to trigger. Stop the alarm and check that the Clock app closes and the Home screen appears.
  • (automated) Create a new alarm with non-default values for the repeat period, alarm sound and minutes to snooze.

Calendar

Metrics

Localization

Marketplace

  • 1332 Marketplace - Verify the user can install market installed hosted app.

Firefox Accounts

  • 13006 Successful Sign up in Settings

Smoke Test Blockers

Bugzilla Query for the Whiteboard tag "smoketest": http://mzl.la/16lkWbM

Escalation Strategy

  • When a Smoke Test Blocker is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
  • The QA Team is notified in the the #fxosqa IRC channel.
  • A regression window for the root cause of the bug is identified.
  • The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
  • The Whiteboard tag "[backout-needed]" is added to the bug.
  • The module owner is also needinfo?ed.
  • A Smoke Test report email is sent daily by the Firefox OS QA team to the following mailing lists: fxosqa-report-watchlist@mozilla.org and b2g-release-drivers@mozilla.org
  • QA tracks the bug until the commit has been backed out.
  • QA requests for a new nightly build to be created.
  • When a new build becomes available, QA verifies that the bug no longer reproduces.

Test Automation

Retrieved from "https://wiki.allizom.org/index.php?title=B2G/QA/Smoke_Tests&oldid=1054122"