B2G/QA/Device Test Plan/NewDevice

Test Plan

Introduction

This test plan is a test plan for testing new devices and some of the things to look out for. This is a guideline; there may be more to test depending on the feature of the phone.

Design Documentation/Background Research

Listed are the related documentation for testing:

Module Owners

• List of Firefox OS Modules/Dev Owners (link) - in case you need to ask questions regarding the new features of the component.
  
• Firefox OS functional teams breakdown (link)
  
• QA owners for Firefox OS components (link)

Requirements:

• Flame
• Woodduck Wiki
• Tarako
• Tako
• Dolphin
• Red Square
• Smart TV, Smart TV 2.5

Bugs tracking features:

• Dashboard - High Priority Bugs (Overall)
• Firefox OS 1.2 Dashboard
• Firefox OS 1.3 Dashboard
• Firefox OS 1.4 Dashboard
• Firefox OS 2.2 Dashboard
• Firefox OS 2.5 Dashboard
• Firefox OS 2.6 Dashboard

Specs:

• See Device Specifications

Tips and Tricks for testing:

• https://wiki.mozilla.org/B2G/QA/Tips_And_Tricks
• https://developer.mozilla.org/en-US/docs/Performance/Profiling_with_the_Built-in_Profiler

OS Diagrams

• https://github.com/sotaroikeda/firefox-diagrams/wiki/Firefox-Diagrams

Hardware Requirements

• A desktop machine capable of running ADB is necessary for testing for getting debug information and flash the device.
• Mobile devices listed
• Bluetooth headset
• headset with microphone
• headset without microphone
• another device/machine with bluetooth transfer capabilities
• Optional: Build machine to build your own gaia/gecko
• Optional: Build OS Storage machine to make it easier to find regressions

Mobile Device

• New Device

Software Requirements

• ADB
• fastboot
• New Device Flashing Tool and USB Driver
• System for Flashing Tool (Typically a Windows Machine)

System Testing

Mobile OS

• Depends on device

Accounts

• See Test Accounts
• Facebook Contact Import accounts also listed in the above link

Email Servers

• Hotmail.com
• Gmail.com
• Mozilla.com

Strategy

• smoke test testing
• target testing for interface between hardware and software
• Exploratory testing

Currently not tested:

• Performance against different mobile data types

Test Deliverables

• Test Results

Dependencies/Risk

The drivers on most devices are given from the OEM vendors. This poses a risk if we do not get an update to a potentially critical bug.

Signoff/Exit Criteria

• no critical bug is found during testing

Test Outline

Note : This test out line is a brain storm of all possible conditions to consider for testing. The scope of the testing will be narrowed from this list based on design information, risks of breakage, etc.

Positive Smoke/Alpha Tests

• Reference Data: Feature Testing

Functional

Smoke Test

Email

• Test email on wifi and mobile data

Gallery

• test various images and various image size (supported/unsupported)

Camera

• test orientation
• test video capture
• test camera/camera preview
• flash
• front camera/back camera

FM Radio

• test audio for stereo output

Music

• music (supported/unsupported encoding)

Video

• video (supported/unsupported encoding)
• streaming video (youtube)

Usage

• Usage Graph

Geolocation

• GPS test app or maps.google.com

Dialer

• make a call in
• make a call out
• conference call
• call waiting
• dual sim support
  • 0 sim, 1 sim, 2 sim
• sim lock
• airplane mode
• sms

tethering

• tethering

Wifi Hotspot

• wifi hotspot

touch screen

• sensitivity

bluetooth

• file transfer
• music control
• dialer

Buttons & Misc

• power button
• volume buttons
• Vibration
• headset
• microphone
• orientation/angle

OTA

• check wifi
• check bluetooth
• check NFC
• check camera
  • front and back
  • check flash
  • check autofocus
• check music
• check geolocation
• check dialer
• check SMS
• check USB storage
• check FM Radio
• check email
• check logshake
• check light sensor ( ie turn on auto brightness and place finger over light sensor )

Beta Conditional / Exploratory Items

• OTA
  • Shallow flashing
    • testing full
  • changing the channel to nightly/aurora, updating
    • testing full
• Physical Conditions
  • Power Consumption
  • Performance
    • Speed of functions
    • Processor utilization
    • Memory utilization
      • Memory leaks?
      • Memory efficiency
  • Storage
    • Disk
    • Sim
• Race Conditions
  • Lock conditions?
  • tooltips or other dialogs?
  • Network Conditions (Should have no impact)
    • WiFi (various supported ranges)
    • Bluetooth (Devices)
    • Mobile Data type :
      • 3G/4G?
      • signal frequency
    • SMS
  • Signal Conflict
  • Battery
    • • Battery Low
      • No Battery
      • While Charged
      • While not Charged
    • Other Apps
      • Calendar notification
      • Alarm Clock notification
      • Sleep notification
      • Screensaver / Lock (system)
      • EnergySaver (Fade) (system)
      • Other?
      • Music playing (should not affect)
      • plugins (youtube and other websites with different media types)
      • Extensions ( addons to fennec)
  • Buttons
    • Hard reset
    • Soft reset
    • Turn off phone
    • force quit
    • Other buttons being pressed on the phone (zoom, picture, power )
  • Endurance Testing

Usability

• Usability
• Accessibility (508 compliance)