WebDriver/Marionette

See also https://wiki.mozilla.org/Platform/JSDebugv2

Problem

We need a new and unified way to be able to automate testing remotely on the B2G environment since we won't have the traditional browser environment to hook onto as we have done in the past, since we'll be booting directly into Gecko and will have to test functionality of the UI. We'll want to be able to test anything the user can do, from entering input, to gestures, etc.

Goals

• We want to be able to send remote testing commands to the mobile device. We do not want to hardcode what kind of test actions are available, we want this to be a unified method of sending commands to the phone, and then having a dispatcher on the mobile side sending the commands to the appropriate listener or harness. This would make this solution robust and able to handle any other test harnesses the future may hold.

• The test actions requested should be in JSON form both for simplicity and eventual support of Selenium / WebDriver commands. Selenium/WebDriver support would entice more developers to write tests due to its popularity and potential as a testing standard.

• If we integrate with remote-debugger, we will be able to send both debugging commands and testing commands over the same socket connection.

• In the first pass, we want to be able to support Eideticker on the remote device, so we should focus support for this project.

• We also wish to integrate WebAPI calls. This would allow us test interactions between two B2G phones with ease. For example, we would like one phone to send an SMS message to the other, and verify the receipt. This would be enabled by supporting WebAPI's WebSMS API.

Design and Implementation

Remote Testing

Note: The current design grew from the ideas presented here. We selected approach#3.

In order to send the JSON commands to the mobile device, we will use the already written socket listener/debugger, remote-debugger. We can integrate our command dispatcher to the remote-debugger in the form of an Actor. This will allow us to connect to the socket and send debug or testing commands by addressing the JSON messages to the appropriate actor. The remote-debugger code is subject to change as it is not yet part of mozilla-central, so the design may change if anything radical happens. It is estimated to land on m-c for the Firefox 11 nightlies, near the Q4's end.

We will have to write our own root actor that will start at Gecko launch. This root actor can then register any number of actors below it. This would allow us to add debugging as well as testing actors. Since the debugging actors currently available are all browser-specific, we may not have any debuggers for Gecko available when Marionette first comes out.

Once the testing actor receives the testing command, it will dispatch it to the appropriate test harness or listener. For now, we plan to use the accessibility (a11y) interfaces that is already supported by the platform. For actions not covered by a11y, like gestures, we should be able to fire off events directly to the mobile device's event queue.

JSON Wire Protocol

The JSON Wire Protocol we will be implementing is here

Development

Tracking Progress

We're currently tracking our progress with Pivotal Tracker. More stories to come.

Notes and Status

Development notes, bugs, status and other neat things are over here

Code

The Marionette code is hosted here.

Marionette and Fennec/Firefox

This is a mercurial patch queue which applies on top of either the remote-debug branch of mozilla-central or mozilla-central itself. I would recommend running it on top of mozilla-central as the remote-debug branch is not always up to date with upstream. In order to apply the patches, first create the patch queue and get the code:

   hg qinit
   cd .hg/patches
   hg clone http://hg.mozilla.org/users/mdas_mozilla.com/marionette-pq .

To apply marionette to mozilla-central, you have to change your series file to the series-add-rd file:

  cp series-add-rd series
  cd ../../
  hg qpush -a

This will import the minimal set of remote-debugger dependencies into your m-c tree and will apply marionette patches to it.

To apply marionette to remote-debug branch:

  cd ../../
  hg qpush -a

By the end of the quarter, both remote-debugger and marionette should land on m-c so we won't need this patch queue for long.

These patches bake marionette into startup of fennec/fireofx, so all you need to do is rebuild and then start up the newly built binary. For both means, the server should be running on port 2828 by default.

Marionette and B2G

To use Marionette with B2G, pull this fork of B2G https://github.com/jonallengriffin/B2G. This tree is kept up to date with upstream often, and will be merged into the standard B2G repo (very) soon.

Marionette is enabled via a pref in gecko/b2g/app/b2g.js. Enable it using the boolean pref "marionette.server.enabled", and set the port it listens to via "marionette.server.port"

Using a client

Once you have the server going, you can use Marionette client to talk to it if you prefer. For an example, look at test_debugger.py.

Writing Tests

We have a marionette test runner included with the Marionette client. This test runner currently only understands python tests driven by the Marionette client, but we will have support for pure JavaScript tests soon.

By default, tests are run in content, but you can set the context of your page to run in chrome as well.

Marionette Client API

For a list of current and future methods supported by the Marionette client, see the Marionette Client API.

Test Structure

Marionette Python tests use the Python unittest framework. Tests belong in methods beginning with 'test' inside a class which is derived from MarionetteTestCase. For example:

   from marionette_test import MarionetteTestCase
   
   class BatteryTest(MarionetteTestCase):
   
       def test1(self):
           # do test steps here

Classes derived from MarionetteTestCase have the following methods and properties:

• self.marionette: a Marionette instance, on which you can call any methods of the Marionette Client API
• self.marionette.emulator: a Marionette Emulator instance, if the tests are being run on an emulator, which allow you to interact directly with the emulator. See https://github.com/jonallengriffin/marionette_client/blob/master/marionette/emulator.py for available emulator methods.
• self.marionette.emulator.battery: an EmulatorBattery instance, for tests running in an emulator, that allows you to interact with the emulator's battery. See https://github.com/jonallengriffin/marionette_client/blob/master/marionette/emulator_battery.py for a list of available methods.
• Python unittest assertion and other methods, documented at http://docs.python.org/library/unittest.html#test-cases

Writing Content Tests

For scripts you are executing in the content space via execute_script and execute_async_script (see the Marionette Client API), if you'd like to access window members that are not part of the nsIDOMWindow interface, then you'll need to access them through wrappedJSObject.

For example, if the page you're on has modified the window like so:

   window.non_standard_member = true;

then you can access this member in your content script by doing:

   window.wrappedJSObject.non_standard_member;

Example Marionette Client Python Tests

This is an example test that talks to a marionette server running on localhost:2828.

from marionette_test import MarionetteTestCase

class UserTest(MarionetteTestCase):
    def test1(self):
        marionette = Marionette("localhost", 2828)
        marionette.start_session()
        marionette.navigate("http://example.com")
        element1 = marionette.find_element("id", "some-element")
        self.assertEqual("some expected text", element1.text)
        element1.click()
        ...etc...

You start the runner by running the following in the marionette/ folder in the marionette_client code base.

   python runtests.py <path to file>

This test currently runs for testing Fennec and Firefox. You can run this test in B2G, but you won't be able to navigate to a page yet.

This test connects with the server, starts a session (which generates a new tab in firefox), opens the given webpage, finds the given element and asserts that the text is as expected and lastly creates a click action on the object. This is an example of how Marionette can be used to drive user actions on a page.

Marionette can also communicate with emulators. The following is a test that communicates with both the emulator, and the code running within it.

from marionette_test import MarionetteTestCase

class BatteryTest(MarionetteTestCase):

    def test(self):
        marionette = self.marionette
        self.assertTrue(marionette.emulator.is_running)

        moz_level = marionette.execute_script("return navigator.mozBattery.level;")
        self.assertEquals(moz_level, marionette.emulator.battery.level)
        marionette.emulator.battery.level = 0.4
        self.assertEquals(0.4, marionette.execute_script("return navigator.mozBattery.level;"))

This test checks the current battery state, then tells the emulator to set the battery to 40 percent capacity, and then verifies that the change is detected by content. This test will run with marionette right now, but we may add helper wrappers for the webAPI calls, so you wouldn't have to inject your own scripts to retrieve the value.

To run emulator tests, you should pass the --emulator and --homedir parameters to runtests.py; --homedir is the B2G homedir. Then runtests.py will launch the emulator and connect to it before running tests.

   python runtests.py --emulator --homedir=/path/to/B2G_HOME <test path>

Alternate Formats for Python Tests

A full of example of a Python test can be found here: https://github.com/jonallengriffin/marionette_client/blob/master/marionette/tests/emulator/test_battery.py

Alternate formats for this same test:

• JS and Python in separate files: http://people.mozilla.org/~jgriffin/test_battery.py and http://people.mozilla.org/~jgriffin/test_battery.js
• JS and Python in same file, non-interleaved: http://people.mozilla.org/~jgriffin/test_battery_single_file.js
• JS and Python in same file, interleaved: http://people.mozilla.org/~jgriffin/test_battery_single_file2.js

Support for these alternate formats aren't yet implemented, but could easily be.

Example JS Tests

You can run pure JavaScript tests as well. These tests have the appearance of mochitests, with support for the "ok" (which is like assertTrue), "is", "isnot" and "finish" commands. You can indicate whether you want these tests to be run in chrome or content by setting key variables (MARIONETTE_CONTEXT), whether the test is async or synchronous (MARIONETTE_ASYNC), and you can set the timeout period if the test script is asynchronous (MARIONETTE_TIMEOUT). These keywords must be defined at the top of the file. Here's an example for testing Gaia's AppManager in B2G:

/* Flags section */
MARIONETTE_CONTEXT = "content"; //keyword describes in which context, chrome or content, to run this script
MARIONETTE_ASYNC = false; //indicates whether the script is async or not
//MARIONETTE_TIMEOUT = 10000; //if it is async, specify a timeout period in ms

/* Test code */
var browserUrl = "../browser/browser.html";
var gaia = window.wrappedJSObject.Gaia;
var runningApps = gaia.AppManager.getRunningApps().length;

//Start the app and assert it is running
gaia.AppManager.launch(browserUrl);
Marionette.is(runningApps + 1, gaia.AppManager.getRunningApps().length);
Marionette.isnot(null, gia.AppManager.getAppInstance(browserUrl));
//kill it
gaia.AppManager.kill(browserUrl);
//verify that we killed it
Marionette.is(runningApps, gaia.AppManager.getRunningApps().length);

//Finish test
Marionette.finish();

This launches the browser and tests if it is running, then closes it.

To run these JS tests with marionette, you'll need to add an ini file defining which files to run. More information will be posted when we have the support for it, but the idea would be to have a section in the .ini file for listing pure marionette/JS tests, and another section to list marionette/python tests. The test runner will then read this .ini file, and run tests accordingly.

Questions/Comments/Concerns

Feel free to drop us a line over at the #ateam channel!