canmove, Confirmed users
2,041
edits
QA/B2G WebAPI Test Plan: Difference between revisions
→Strategy
(→Scope) |
|||
| Line 23: | Line 23: | ||
== Strategy == | == Strategy == | ||
Functional test strategy will vary between APIs based largely on how dependent they are on actual device hardware. B2G continuous integration will offer two options: [https://developer.mozilla.org/en/Mochitest mochitest-plain] and a new [[Auto-tools/Projects/Marionette|Marionette]]-based JS harness. | Functional test strategy will vary between APIs based largely on how dependent they are on actual device hardware and whether API effects can be verified via automation. Broadly, testing will be divided between continuous integration-driven isolated API tests and test web apps to manipulate APIs directly on-device. | ||
=== CI-driven API Testing === | |||
B2G continuous integration will offer two options: [https://developer.mozilla.org/en/Mochitest mochitest-plain] and a new [[Auto-tools/Projects/Marionette|Marionette]]-based JS harness. | |||
Tests written against the new harness will run only in B2G CI against QEMU, a device emulator. They can set things like orientation or battery status to a known value, which allows for rich virtual device manipulation for fixture setup. | Tests written against the new harness will run only in B2G CI against QEMU, a device emulator. They can set things like orientation or battery status to a known value, which allows for rich virtual device manipulation for fixture setup. | ||
Tests written against mochitest-plain can run in the standard mozilla-central [https://tbpl.mozilla.org/ Tinderbox] test environment and anywhere else mochitests are typically run, as well as against the B2G codebase. Mochitest will give wider coverage, but will not be able to use the emulator to set up device-related fixtures and is thus significantly more limited. | Tests written against mochitest-plain can run in the standard mozilla-central [https://tbpl.mozilla.org/ Tinderbox] test environment and anywhere else mochitests are typically run, as well as against the B2G codebase. Mochitest will give wider coverage, but will not be able to use the emulator to set up device-related fixtures and is thus significantly more limited. | ||
=== Test Web Apps === | |||
Some APIs have effects than cannot be verified through unattended automation. One example would be whether a microphone is open and receiving sound. Another example would be whether the phone is actually vibrating in response to an API call. | |||
For these APIs, we must test using web apps designed to exercise the APIs in a known fashion such that the effects can be manually verified. | |||
It would be tempting to use Gaia for this, but Gaia apps cannot be guaranteed to be using the APIs in a given way; they might refactor in the future. So, part of this testing project will be implementing apps whose API usage can be absolutely controlled. | |||
It is still undetermined whether these apps will be implemented as web pages | |||