MozillaWiki

B2G/QA/WebAPI Test Plan: Difference between revisions

< B2G‎ | QA
(Created page with "=Boot2Gecko WebAPI Test Plan= == Summary == {| class="fullwidth-table" |- | style="width:20%" | '''Lead''' | [mailto:gmealer@mozilla.com Geo Mealer] (irc: geo) |- | '''Cont...")
 
No edit summary
 
(56 intermediate revisions by 6 users not shown)
Line 1: Line 1:
=OBSOLETE=
Please note that this test plan is now obsolete, and is no longer underway.
=Boot2Gecko WebAPI Test Plan=
=Boot2Gecko WebAPI Test Plan=


Line 8: Line 12:
  |-
  |-
  | '''Contributors'''
  | '''Contributors'''
  | TBD
  | [mailto:mwargers@mozilla.com Martijn Wargers] (irc: mw22)<br />
[mailto:dclarke@mozilla.com David Clarke] (irc: onecyrenus)<br />
[mailto:jsmith@mozilla.com Jason Smith] (irc: jsmith)
  |-
  |-
  | '''Status'''
  | '''Status'''
  | Pre-release, approaching [https://docs.google.com/spreadsheet/ccc?key=0AiBigu584YY7dGlNSlY0QzhJb3M5anRBa1gxalV0Y3c#gid=0 Milestone 3]
  | Pre-release, approaching [https://docs.google.com/spreadsheet/ccc?key=0AiBigu584YY7dGlNSlY0QzhJb3M5anRBa1gxalV0Y3c#gid=0 Milestone 5]
|-
| '''Tracker'''
| [https://www.pivotaltracker.com/projects/578531 Pivotal Project]
  |-
  |-
  | '''Project Page'''
  | '''Project Page'''
  | [[WebAPI]]
  | [[WebAPI]]
|-
| '''References'''
| [[/Instructions for Contributors|Instructions for Contributors]]
  |}
  |}


Line 23: Line 35:


== Strategy ==
== Strategy ==
Functional test strategy will vary between APIs based largely on how dependent they are on actual device hardware and whether API effects can be verified via automation. Broadly, testing will be divided between continuous integration-driven isolated API tests and test web apps to manipulate APIs directly on-device.
Functional test strategy will vary between APIs based on whether API effects can be effectively verified via automation, and whether that automation requires an emulator or not.


=== CI-driven API Testing ===
Broadly, testing will be divided between automation executed in continuous integration, and interactive exercises designed to manipulate APIs directly on-device. The expected amount of time given to each is roughly 75% to planning and CI implementation and 25% to implementing interactive exercises. However, as the exercises will be necessary for final qualification of the B2G stack on-device, those priorities may change closer to the shipping date.
 
=== CI-driven Automation ===
B2G continuous integration will offer two options: [https://developer.mozilla.org/en/Mochitest mochitest-plain] and a new [[Auto-tools/Projects/Marionette|Marionette]]-based JS harness.  
B2G continuous integration will offer two options: [https://developer.mozilla.org/en/Mochitest mochitest-plain] and a new [[Auto-tools/Projects/Marionette|Marionette]]-based JS harness.  


Line 32: Line 46:
Tests written against mochitest-plain can run in the standard mozilla-central [https://tbpl.mozilla.org/ Tinderbox] test environment and anywhere else mochitests are typically run, as well as against the B2G codebase. Mochitest will give wider coverage, but will not be able to use the emulator to set up device-related fixtures and is thus significantly more limited.
Tests written against mochitest-plain can run in the standard mozilla-central [https://tbpl.mozilla.org/ Tinderbox] test environment and anywhere else mochitests are typically run, as well as against the B2G codebase. Mochitest will give wider coverage, but will not be able to use the emulator to set up device-related fixtures and is thus significantly more limited.


=== Test Web Apps ===
=== Interactive Exercises ===
Some APIs have effects than cannot be verified through unattended automation. One example would be whether a microphone is open and receiving sound. Another example would be whether the phone is actually vibrating in response to an API call.
Some APIs have effects that cannot be verified through unattended automation. One example would be whether a microphone is open and receiving sound. Another example would be whether the phone is actually vibrating in response to an API call. These APIs must be verified interactively.


For these APIs, we must test using web apps designed to exercise the APIs in a known fashion such that the effects can be manually verified.  
It would be tempting to use Gaia to exercise the API, but Gaia apps cannot be guaranteed to be using the APIs in a controlled fashion; that would require inappropriately locking in their implementation. So for these APIs, it is best to test using code specifically designed to exercise the APIs in a known fashion. In addition, as CI automation cannot be currently run on a final target device, it may be useful to have exercises even for APIs we address primarily through CI.


It would be tempting to use Gaia for this, but Gaia apps cannot be guaranteed to be using the APIs in a given way; they might refactor in the future. So, part of this testing project will be implementing apps whose API usage can be absolutely controlled.
One decision is whether these exercises will be implemented as web pages or installable apps. Apps would be closer to how a developer will mostly use these APIs, and some APIs may only work for standalone applications. However, pages are more convenient as they don't require being preloaded onto the phone before testing can occur; they can just be navigated via onboard browser.
 
Tentatively, the plan is to implement exercises as pages where possible for expediency's sake, but to eventually move them into a standalone app that developers and the testing community can experiment with.
 
Also see the [[/On_Device|On Device Test Plan]].
 
== API Sub-Plan List ==
 
{| class="fullwidth-table"
|-
| style="width:20%" | '''API'''
| style="width:42%" | '''Description'''
| style="width:20%" | '''API Status'''
| style="width:18%" | '''Automation Status'''
|-
| [[/Screen Orientation|Screen Orientation]]
| Screen Orientation notifications and locking
| Ready
| [https://bugzilla.mozilla.org/show_bug.cgi?id=760735 First round done]
|-
| [[/Web Activities|Web Activities]]
| Delegate an activity to another application.
| Ready
| Developer contacted
|-
| [[/Vibration|Vibration]]
| Control device vibration for things like haptic feedback in games.
| Ready
| Developer contacted [https://bugzilla.mozilla.org/show_bug.cgi?id=679966 First round done by developer]
|-
| [[/Alarm|Alarm]]
| Schedule a notification, or for an application to be started, at a specific time.
| Ready
| Tests in Suite
|-
| [[/Push Notifications|Push Notifications]]
| Allow the platform to send notification messages to specific applications.
| Unknown (TEF)
| Not Started
|-
| [[/Geolocation|Geolocation]]
| Get current device position
| Ready
| Developer contacted
|-
| [[/Device Storage|Device Storage]]
| Add/Read/Modify files stored on a central location on the device.
| Ready
| First Round Completed, Blocked on Outstanding Bug
|-
| [[/Contacts|Contacts]]
| Add/Read/Modify the device contacts address book.
| Ready (Priv)
| Not Started
|-
| [[/Camera|Camera]]
| Control of front/rear cameras on-device
| Unknown
| Not Started
|-
| [[/TimeClock|Time/Clock]]
| Get notifications about time changes (high priority). Set current time (low priority).
| Ready
| Not Started
|-
| [[Idle]]
| Get notifications when user is idle.
| Ready
| Not Started
|-
| [[/Battery Status|Battery Status]]
| Information about battery charge level and if device is plugged in.
| Ready
| Not Started
|-
| [[/FM Radio|FM Radio]]
| For FM radio feature.
| In Progress (Priv)
| Not Started
|-
| [[/WebSMS|WebSMS]]
| Send/receive SMS messages as well as manage messages stored on device. 
| Ready (Priv)
| Planning
|-
| [[/Permissions|Permissions]]
| Allow an app to manage all app permissions in a centralized location
| In Progress (Priv)
| Not Started
|-
| [[/WebTelephony|WebTelephony]]
| Allow placing and answering phone calls as well as build in-call UI.
| Ready (Priv)
| Not Started
|-
| [[/Settings|Settings]]
| Set system-wide configurations that are saved permanently on the device.  
| Ready (Priv)
| Not Started
|-
| [[/Browser|Browser]]
| Enables implementing a browser completely in web technologies.
| Ready (Priv)
| Not Started
|-
| [[/Web Bluetooth|Web Bluetooth]]
| Low level access to Bluetooth hardware. (Currently headset-only)
| Ready (Priv)
| Not Started
|-
| [[/Mobile Connection|Mobile Connection]]
| Expose signal strength, operator, etc for GSM and other mobile connections.
| Ready (Priv)
| Not Started
|-
| [[Power Management|Power Management]]
| Turn on/off screen, cpu, device power, etc. Listen and inspect resource lock events.
| Ready (Priv)
| Tests in Suite
|-
| [[/WiFi Info|Wifi Info]]
| Enumerate WiFi networks, get signal strength and name of current network, etc.  
| Ready
| Not Started
|-
| [[/WebPayment|WebPayment]]
| Allow payments for virtual goods
| Ready
| Unit test automation present, Gaia UI Test automation currently being investigated
|}


It is still undetermined whether these apps will be implemented as web pages or web apps. Apps would be closer to how a developer will mostly use these APIs. Pages don't require being preloaded onto the phone before testing can occur; they can just be navigated via onboard browser.
<small>Template for new Sub-Plans can be found [[B2G/QA/WebAPI Test Plan/Template|here]].</small>

Latest revision as of 17:24, 30 March 2015

OBSOLETE

Please note that this test plan is now obsolete, and is no longer underway.

Boot2Gecko WebAPI Test Plan

Summary

Lead Geo Mealer (irc: geo)
Contributors Martijn Wargers (irc: mw22)

David Clarke (irc: onecyrenus)
Jason Smith (irc: jsmith)

Status Pre-release, approaching Milestone 5
Tracker Pivotal Project
Project Page WebAPI
References Instructions for Contributors

Scope

This test plan covers qualification of WebAPIs developed for or used specifically in Boot2Gecko to interface with mobile devices. It does not cover the entirety of Gecko APIs available. The individual APIs covered are listed below and broken out into their own individual sub-plans.

Currently, only functional testing is being planned. Ultimately, performance and security should be addressed. However, there are too many unknowns at time of writing to adequately plan for them in advance.

Strategy

Functional test strategy will vary between APIs based on whether API effects can be effectively verified via automation, and whether that automation requires an emulator or not.

Broadly, testing will be divided between automation executed in continuous integration, and interactive exercises designed to manipulate APIs directly on-device. The expected amount of time given to each is roughly 75% to planning and CI implementation and 25% to implementing interactive exercises. However, as the exercises will be necessary for final qualification of the B2G stack on-device, those priorities may change closer to the shipping date.

CI-driven Automation

B2G continuous integration will offer two options: mochitest-plain and a new Marionette-based JS harness.

Tests written against the new harness will run only in B2G CI against QEMU, a device emulator. They can set things like orientation or battery status to a known value, which allows for rich virtual device manipulation for fixture setup.

Tests written against mochitest-plain can run in the standard mozilla-central Tinderbox test environment and anywhere else mochitests are typically run, as well as against the B2G codebase. Mochitest will give wider coverage, but will not be able to use the emulator to set up device-related fixtures and is thus significantly more limited.

Interactive Exercises

Some APIs have effects that cannot be verified through unattended automation. One example would be whether a microphone is open and receiving sound. Another example would be whether the phone is actually vibrating in response to an API call. These APIs must be verified interactively.

It would be tempting to use Gaia to exercise the API, but Gaia apps cannot be guaranteed to be using the APIs in a controlled fashion; that would require inappropriately locking in their implementation. So for these APIs, it is best to test using code specifically designed to exercise the APIs in a known fashion. In addition, as CI automation cannot be currently run on a final target device, it may be useful to have exercises even for APIs we address primarily through CI.

One decision is whether these exercises will be implemented as web pages or installable apps. Apps would be closer to how a developer will mostly use these APIs, and some APIs may only work for standalone applications. However, pages are more convenient as they don't require being preloaded onto the phone before testing can occur; they can just be navigated via onboard browser.

Tentatively, the plan is to implement exercises as pages where possible for expediency's sake, but to eventually move them into a standalone app that developers and the testing community can experiment with.

Also see the On Device Test Plan.

API Sub-Plan List

API Description API Status Automation Status
Screen Orientation Screen Orientation notifications and locking Ready First round done
Web Activities Delegate an activity to another application. Ready Developer contacted
Vibration Control device vibration for things like haptic feedback in games. Ready Developer contacted First round done by developer
Alarm Schedule a notification, or for an application to be started, at a specific time. Ready Tests in Suite
Push Notifications Allow the platform to send notification messages to specific applications. Unknown (TEF) Not Started
Geolocation Get current device position Ready Developer contacted
Device Storage Add/Read/Modify files stored on a central location on the device. Ready First Round Completed, Blocked on Outstanding Bug
Contacts Add/Read/Modify the device contacts address book. Ready (Priv) Not Started
Camera Control of front/rear cameras on-device Unknown Not Started
Time/Clock Get notifications about time changes (high priority). Set current time (low priority). Ready Not Started
Idle Get notifications when user is idle. Ready Not Started
Battery Status Information about battery charge level and if device is plugged in. Ready Not Started
FM Radio For FM radio feature. In Progress (Priv) Not Started
WebSMS Send/receive SMS messages as well as manage messages stored on device. Ready (Priv) Planning
Permissions Allow an app to manage all app permissions in a centralized location In Progress (Priv) Not Started
WebTelephony Allow placing and answering phone calls as well as build in-call UI. Ready (Priv) Not Started
Settings Set system-wide configurations that are saved permanently on the device. Ready (Priv) Not Started
Browser Enables implementing a browser completely in web technologies. Ready (Priv) Not Started
Web Bluetooth Low level access to Bluetooth hardware. (Currently headset-only) Ready (Priv) Not Started
Mobile Connection Expose signal strength, operator, etc for GSM and other mobile connections. Ready (Priv) Not Started
Power Management Turn on/off screen, cpu, device power, etc. Listen and inspect resource lock events. Ready (Priv) Tests in Suite
Wifi Info Enumerate WiFi networks, get signal strength and name of current network, etc. Ready Not Started
WebPayment Allow payments for virtual goods Ready Unit test automation present, Gaia UI Test automation currently being investigated

Template for new Sub-Plans can be found here.

Retrieved from "https://wiki.allizom.org/index.php?title=B2G/QA/WebAPI_Test_Plan&oldid=1065209"