B2G/QA/Smoke Tests

High Severity Bugs

• Any action that bricks the device.
• Any action that causes the device not to come up again after reboot.
• Any action that causes the device to crash and reboot.
• The following Sanity Test Cases fail without a workaround:
  • [RIL] Correct Radio Interface Layer is installed
  • [OTA] Over The Air update is successful
  • [Homescreen] The Homescreen is viewable
  • [Dialer] Can make an Outbound Call twice
  • [Dialer] Can make an Inbound Call twice
  • [SMS] Can send an Outbound Message twice
  • [SMS] Can receive an Inbound Message twice
  • [Camera] Can take a photo
  • [Camera] Can view the photo taken
  • [Browser] Can browse for 1-2 minutes over WiFi

Note: The telephony tests are done twice to make sure that the process cleans up after itself when the first instance is over so that the second instance is also successful.

Google Spreadsheet that tracks daily test logs for Sanity Test Cases.

Escalation Strategy

• When a High Severity Bug is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
• A notification email is sent by QAnalysts to the following mailing lists: mozilla-dev-b2g@lists.mozilla.org, dev-b2g@lists.mozilla.org, dev-gaia@lists.mozilla.org and b2g-release-drivers@mozilla.org
• The IRC channel owners of #b2g and #gaia are notified in their respective channels.
• A bug is filed requesting OTA to be blocked.
• A regression window for the root cause of the bug is identified.
• The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
• The Whiteboard tag "[backout-needed]" is added to the bug.
• The module owner is also needinfo?ed.
• QA tracks the bug until the commit has been backed out. Bugzilla Query for the Whiteboard tag "[backout-needed]": http://mzl.la/1v2bcJJ
• QA requests for a new nightly build to be created.
• When a new build becomes available, QA verifies that the bug no longer reproduces.
• If the bug no longer reproduces, the bug requesting OTA to be reopened and a request is made to enable OTA again.

Smoke Tests by Functional Area

Google Spreadsheet that tracks daily test logs for Smoke Test Cases.

Communications

• QA Owner: Johan Lorenzo & Eric Chang
• Bugzilla Components Watched:
  • Eric:
  • Johan: Gaia::Dialer, Gaia::Contacts

Dialer

• 1298 Verify an outbound call can be made and that the Call Volume can be adjusted
• 6076 Verify that touch tones work using Mozilla's conference line.
• (automated) Make an outbound call
• (automated) Receive an incoming call with locked screen.

Messages

• 10742 Send an MMS with picture attached

SIM

• 14361 Dialer Send SIM control codes for pin change
• 10739 Send MMS to secondary SIM card (Auto retrieve on with roaming)

Contacts

• 1309 (automated) Add new contact
• 1310 (automated) Edit a contact
• 1314 (automated) Send an SMS to a contact
• 1309 (automated) Add photo from Gallery to contact

Emergency Call

• 15186 (automated) A regular phone number should not be dialed out on the emergency dialer

Telephony/Networking

WiFi

• 6075 Verify the user can connect to WPA/PSK & WPA2/AES WiFi access points
• 1327 (automated) Load a website via WiFi connection

Cellular Data

• 1328 (automated) Load a website via Cellular Data connection

Cost Control

• 8938 (automated) Check if the data alert is triggered after 1-2 minutes of browsing over cellular data

Bluetooth

• 6071 Verify that the user can enable Bluetooth and pair to a Bluetooth enabled computer/another device.
• 6070 Pair and receive an incoming call using bluetooth
• 9346 Music can be played and stopped remotely by user
• 6071 (automated) Verify that the user can enable Bluetooth via Settings, rename the device and is discoverable by other bluetooth-enabled devices nearby.

FM Radio

• 2461 (automated) Verify that headphones are plugged in and the FM radio has tuned in to the default channel frequency (lower bound). Flick up the frequency dialer a few times and check that the FM radio has tuned in to a higher default frequency.

Media

• QA Owner: No-Jun Park
• Bugzilla Components Watched: Gaia::Camera, Gaia::Video, Gaia::Music, Gaia::Gallery

Camera

• Bugzilla Component: Gaia::Camera

• 2460 Verify the user can take a picture with camera from lockscreen.
• 14789 Take pictures using Face recognition and Tap to Focus
• 2477 (automated) Launch the Camera app, switch to video mode, record 3 seconds of video, check that the video has been saved on the SD Card
• 1325 (automated) Launch the Camera app, take a photo, check that the thumbnail is visible and that the photo has been saved on the SD Card. Repeat this action twice more.

Video

• Bugzilla Component: Gaia::Video
• 2478 Play the video you recorded. Check for video and sound

Music

• Bugzilla Component: Gaia::Music
• 4031 Artist View - Verify the user can play a single song.

Gallery

• Bugzilla Component: Gaia::Gallery
• 14645 Test you can view a thumbnail in large mode
• 14645 (automated) Check that photos can be viewed in portrait and landscape mode if the phone orientation is changed
• 1326 (automated) Tap first image to open full screen view, then flick forward and backward to navigate.

Systems Front End

• QA Owner: Naoki Hirata / Parul Mathur
• Bugzilla Components Watched: Gaia::Browser, Gaia::First Time Experience, Gaia::Homescreen, Gaia::Search, Gaia::System, Gaia::System::Browser Chrome, Gaia::System::Window Mgmt

Browser

• 1327 (automated) Can browse over WiFi
• 1328 (automated) Can browse over cell data
• 6073 (automated) Play a video, pause, resume, mute, view in full screen

Homescreen

• 14644 Battery Charge indicator given when charging
• 6117 Verify the user can delete a packaged app
• 1333 Marketplace - Verify the user can launch market installed hosted apps.
• 5856 Install and launch Facebook and Twitter apps from Homescreen Search
• 6116 Install and launch a packaged app
• 6786 Verify a market installed hosted app can be installed and removed.
• (automated) Install an app and verify that its icon appears on the homescreen. Activate the edit mode, tap on the (x) to start delete process, and tap on the confirm delete button. Check that the app is no longer available.
• 6117 (automated) Install a packaged app and verify that its icon appears on the homescreen. Activate the edit mode, tap on the (x) to start delete process, and tap on the confirm delete button. Check that the app is no longer available.
• (automated) Install an app and verify that its icon appears on the homescreen. Launch the app and verify that it is visible.
• 6116 aster/tests/python/gaia-ui-tests/gaiatest/tests/functional/homescreen/test_homescreen_launch_app_packaged.py (automated) Install a packaged app and verify that its icon appears on the homescreen. Launch the app and verify that it is visible.
• 1317 (automated) Move an app. Verify that the move activates the Edit mode. Exit the Edit mode and verify that the app has moved into the correct position.

Rocketbar

• (automated) On the homescreen, bring up the context menu and select Add Collection. Verify that the collection is added to the homescreen. Tap the collection and select an app to save to the homescreen. Verify that the app was added to the homescreen.

Task Manager

• 2462 (automated) Launch the Clock and Calendar apps. Long press the Home button to get the cards view and swipe to the previous app. Tap the card view of the app to bring it up again.

First Time Use

• 6119 (automated) Launch the First Time Use tour, select en-US as the language, enable data if cellular network is present, connect to WiFi, set timezone if no cellular network, disable Geolocation, import contacts from SIM Card, skip Firefox Accounts, enable telemetry data collection, enter a dummy email address and skip the device tour.
• 6119 (automated) Go through FTU with default values and walk through the tour. Check that you can go back a step and then continue till the tour is finished.
• 6119 (automated) Go through FTU and verify that the Status bar is displayed throughout.

Graphics

• QA Owner: No-Jun Park
• Bugzilla Components Watched: Canvas: 2D, Canvas: WebGL, GFX: Color Management, Graphics, Graphics: Layers, Image Blocking, ImageLib, Layout, Layout: Block and Inline, Layout: Floats, Layout: Form Controls, Layout: HTML Frames, Layout: Images, Layout: Misc Code, Layout: R & A Pos, Layout: Tables, Layout: Text, Layout: View Rendering, Panning and Zooming, Video/Audio
• Currently there is no graphics specific test cases in the smoketest suite- if a smoketest failure occurs with graphical issues, it will be considered as a graphic smoketest blocker bug.

Systems Platform

Lockscreen

• 15243 Verify Lockscreen Display after timeout
• 6784 (automated) Lock the device and then unlock it to homescreen. Verify that the homescreen is displayed.
• 2460 (automated) Turn off the Camera geolocation prompt. Lock the device and then unlock it to camera. Verify that the Camera app is displayed.
• 2460 (automated) Turn off the Camera geolocation prompt. Set a passocde for the lockscreen and enable it. Lock the device and then unlock it to camera. Verify that the passcode is needed. Verify that the Camera app is displayed. Take a photo and verify that it is stored on the SD Card.

OTA

• 2313 System Updates - Success over Wifi
• 14028 Verify Settings Data is Preserved after OTA
• 14029 Verify Homescreen Data is Preserved after OTA
• 14030 Verify Marketplace Data is Preserved after OTA
• 14039 Verify Contact Data is Preserved after OTA
• 14040 Verify Music Data is Preserved after OTA
• 14041 Verify FM Radio Data is Preserved after OTA
• 14042 Verify Notification Data is Preserved after OTA
• 14043 Verify Usage app Data is Preserved after OTA
• 14044 Verify Browser Data is Preserved after OTA
• 14045 Verify Email Data is Preserved after OTA
• 14051 Verify Gallery Data is Preserved after OTA
• 14052 Verify Calendar Data is Preserved after OTA
• 14053 Verify Clock Data is Preserved after OTA
• 14054 Verify Messaging Data is Preserved after OTA

Multimedia Platform

Storage

• 6072 Verify USB Mass Storage Functionality is correct

Productivity

Email

• 10744 Notification for E-mail will be display
• (automated) Connect to WiFi. Launch the email app. Setup a basic Gmail account with name, email address and password. Verify that the email app's header and toolbar area are visible. Verify that the account has emails after it has synced.
• 6113 6114 (automated) Connect to WiFi. Launch the email app. Setup an IMAP account. Verify that the email app's header and toolbar area are visible. Verify that the account has emails after it has synced. Send to an email to oneself, wait till the new email is received and verify that it has the expected subject.
• 10744 (automated) Connect to WiFi. Launch the email app. Setup an IMAP account. Wait for the account to sync. Exit the email app. Send an email to this IMAP account from a separate email client. Wait for the email notification to be displayed. Expand the notification bar and tap the notification. Wait for the email app to launch and verify that the email has been received.

Clock

• 1772 [Alarm] Test that when the alarm triggers, the phone vibrates and an audible alarm is triggered
• 1772 1775 (automated) Create a new alarm, give it a label and save it. Check that the confirmation message appears with a countdown. Edit the alarm and set it to trigger 1 minute from the current time. Turn off the alarm and turn it back on. Press the Home button and wait for the alarm to trigger. Stop the alarm and check that the Clock app closes and the Home screen appears.

Calendar

• 6118 (automated) Can create a new event and view it on all calendar views: month, week, day

Marketplace

• 1332 Marketplace - Verify the user can install market installed hosted app.
• (automated) Verify that the Marketplace icon exists on the homescreen. Launch the Marketplace app and verify that it is displayed.
• (automated) Launch the Marketplace app, search for a privileged app, install it and verify its icon is present on the homescreen.

Firefox Accounts

• 13006 Successful Sign up in Settings

Smoke Test Blockers

Bugzilla Query for the Whiteboard tag "smoketest": http://mzl.la/16lkWbM

Escalation Strategy

• When a Smoke Test Blocker is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
• The QA Team is notified in the the #fxosqa IRC channel.
• A regression window for the root cause of the bug is identified.
• The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
• The Whiteboard tag "[backout-needed]" is added to the bug.
• The module owner is also needinfo?ed.
• A Smoke Test report email is sent daily by the Firefox OS QA team to the following mailing lists: fxosqa-report-watchlist@mozilla.org and b2g-release-drivers@mozilla.org
• QA tracks the bug until the commit has been backed out.
• QA requests for a new nightly build to be created.
• When a new build becomes available, QA verifies that the bug no longer reproduces.

Test Automation

• Gaia UI Tests: https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests