B2G/QA/Smoke Tests

High Severity Bugs

• Any action that bricks the device.
• Any action that causes the device not to come up again after reboot.
• Any action that causes the device to crash and reboot.
• The following Sanity Test Cases fail without a workaround:
  • [RIL] Correct Radio Interface Layer is installed
  • [OTA] Over The Air update is successful
  • [Homescreen] The Homescreen is viewable
  • [Dialer] Can make an Outbound Call twice
  • [Dialer] Can make an Inbound Call twice
  • [SMS] Can send an Outbound Message twice
  • [SMS] Can receive an Inbound Message twice
  • [Camera] Can take a photo
  • [Camera] Can view the photo taken
  • [Browser] Can browse for 1-2 minutes over WiFi

Note: The telephony tests are done twice to make sure that the process cleans up after itself when the first instance is over so that the second instance is also successful.

Google Spreadsheet that tracks daily test logs for Sanity Test Cases.

Escalation Strategy

• When a High Severity Bug is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
• A notification email is sent by QAnalysts to the following mailing lists: mozilla-dev-b2g@lists.mozilla.org, dev-b2g@lists.mozilla.org, dev-gaia@lists.mozilla.org and b2g-release-drivers@mozilla.org
• The IRC channel owners of #b2g and #gaia are notified in their respective channels.
• A bug is filed requesting OTA to be blocked.
• A regression window for the root cause of the bug is identified.
• The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
• The Whiteboard tag "[backout-needed]" is added to the bug.
• The module owner is also needinfo?ed.
• QA tracks the bug until the commit has been backed out. Bugzilla Query for the Whiteboard tag "[backout-needed]": http://mzl.la/1v2bcJJ
• QA requests for a new nightly build to be created.
• When a new build becomes available, QA verifies that the bug no longer reproduces.
• If the bug no longer reproduces, the bug requesting OTA to be blocked is updated.

Smoke Tests by Functional Area

Smoke Test Blockers

Bugzilla Query for the Whiteboard tag "smoketest": http://mzl.la/16lkWbM

Escalation Strategy

• When a Smoke Test Blocker is found, it is reported in Bugzilla with the Whiteboard tag "smoketest".
• The QA Team is notified in the the #fxosqa IRC channel.
• A regression window for the root cause of the bug is identified.
• The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
• The Whiteboard tag "[backout-needed]" is added to the bug.
• The module owner is also needinfo?ed.
• A Smoke Test report email is sent daily by the Firefox OS QA team to the following mailing lists: fxosqa-report-watchlist@mozilla.org and b2g-release-drivers@mozilla.org
• QA tracks the bug until the commit has been backed out.
• QA requests for a new nightly build to be created.
• When a new build becomes available, QA verifies that the bug no longer reproduces.

Test Automation

• Gaia UI Tests: https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests