Mobile/Fennec/Android/Testing

This page has instructions for running tests locally on a device (Android phone, tablet, or emulator) of your choice.

Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.

For the impatient...

mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:

 mach robocop
 mach mochitest
 mach reftest
 mach crashtest
 mach jstestbrowser
 mach xpcshell-test
 mach cppunittest

They will all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.

Test Environment

When testing Firefox for Android with make or mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:

• a "host" computer, running Linux or OSX
• a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer
• a Firefox for Android build, including an apk
• "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform
• a "device manager" to communicate with the Android device
• a TCP/IP network connection between host and device

A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.

Running tests with mach simplifies environment concerns significantly:

• If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.
• If Firefox for Android is not installed on the device, mach will offer to install Firefox.
• If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.

robocop

General Robocop information and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.

The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.

To get tests running locally, first build and install Firefox for Android,

 mach build
 mach package
 mach install

Next, execute mach robocop which installs the Robocop APK to the device and starts testing the entire test suite.

 mach robocop

or

 mach robocop <test-name>

If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall.

 mach build mobile/android/tests/browser/robocop
 mach robocop

This builds the tests in mobile/android/tests/browser/robocop and then installs the debug-signed Robocop APK onto the device. (Building mobile/android also builds the tests within mobile/android/tests.)

Notes:

• Mach is self documenting! For help, try mach help robocop.
• To run one test at a time, find the test name (like "testLoad") in mobile/android/tests/browser/robocop/robocop.ini and pass it as an argument, like: mach robocop testLoad.
• A rooted device is NOT required.

Troubleshooting:

• Ensure the device's screen is on
• Ensure that the device and host machine are on the same network.
  • Are the phone and the desktop both using wifi? (wifi vs ethernet??)
  • Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)
  • Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.
• Ensure Robocop has the correct IP address for your machine
  • Make sure _SERVER_ADDR in the Robocop output is the same as your machine's IP address.
• If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
• Additional tips at Auto-tools/Projects/Robocop#Frequently_found_errors

mochitest (plain and chrome)

General mochitest info General mochitest-chrome info

Pre-requisites:

• Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install
• Ensure your device is connected and visible with "adb devices".
• Ensure that the device and host machine are on the same network <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is not sufficient.
• Ensure that MOZ_HOST_BIN has been set according to the directions above.

Running tests:

 mach mochitest -f plain|chrome
 OR mach mochitest -f plain|chrome <test-dir>
 OR mach mochitest -f plain|chrome <test-dir>/<test-name>

Use "find -name mochitest.ini" to find valid test directories for mochitest plain.

Use "find -name chrome.ini" to find valid test directories for mochitest chrome.

Notes:

• This mach command only supports the adb device manager.

xpcshell

General xpcshell test information.

Be sure you have successfully built Firefox for Android and generated an APK, as described above.

To run all tests referenced by the master xpcshell manifest:

 mach xpcshell-test

(Currently the master manifest is restricted: not all tests are run.)

To run a subset of tests in the specified directory:

 mach xpcshell-test <test-directory>
 OR mach xpcshell-test <test-directory>/<test-name>

Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:

 mach xpcshell-test <test-directory> --no-setup
 OR mach xpcshell-test <test-directory>/<test-name> --no-setup

Notes:

• A rooted device IS required.
• Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.

cppunittests

General cppunit test information

To run a single compiled code test:

 mach cppunittest <test>

For example,

 mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers

To run all the compiled code tests listed in the master cppunittests.ini manifest:

 mach cppunittest

Notes:

• Specifying the test by relative path is difficult; specify an absolute path to the test binary.
• It is not possible to run all tests in a directory.
• All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.

reftests (and crashtests and js-reftests)

General reftest information.

Running reftests:

 mach reftest
 OR mach reftest <test-dir>
 OR mach reftest <test-dir>/<test-name>

Use "find -name reftest.list" to find valid test directories.

Running crashtests:

 mach crashtest
 OR mach crashtest <test-dir>
 OR mach crashtest <test-dir>/<test-name>

Use "find -name crashtest.list" to find valid test directories.

Running js-reftests:

 mach jstestbrowser

Notes:

• There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).

browser-chrome

Before you run tests, you will need to make sure you have packaged the tests in your object dir:

 make -C <objdir-droid> package-tests

There is currently no special make command to build and run browser-chrome tests, but it should be possible to make them run by calling:

 cd <objdir-droid>/_tests/testing/mochitest
 python runtestsremote.py --dm_trans=adb --test-path=mobile --browser-chrome --deviceIP=1.2.3.4 
                          --app=org.mozilla.fennec_$USER --xre-path=<objdir_x86>/dist/bin/

Running tests on the Android emulator

The "Android 2.3 API9 opt", "Android 4.3 API11+ opt" and "Android 4.3 API11+ Debug" tests on treeherder run in an Android ARM emulator. "Android 4.2 x86 opt" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: Running the same tests on the same emulator on different host hardware may produce different results. You can also borrow a slave: https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_a_slave.

Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is now quite simple:

 ./mach android-emulator

By default, the 'android-emulator' command will download the Android 4.3 API11+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)

To use the Android 2.3 API9 image:

 ./mach android-emulator --version 2.3

To use the Android 4.2 x86 image:

 ./mach android-emulator --version x86

The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be much faster.

If you want to "reset" an image (throw away any installed apks and/or settings changes):

 ./mach android-emulator --force-update

will discard the previous image and download a new one.

Once an emulator is running, you can run tests against it just like any other Android device. For example:

 ./mach android-emulator && ./mach install && ./mach mochitest

For the very lazy, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:

 $ ./mach mochitest testing/mochitest/tests/Harness_sanity
 No Android devices connected. Start an emulator? (Y/n) y
 Starting emulator running Android 4.3...
 It looks like Firefox is not installed on this device.
 Install Firefox? (Y/n) y
 Installing Firefox. This may take a while...
 From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.
 ######
 ### Now running mochitest-plain.
 ######
  0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests
  0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...
  0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...
  0:01.82 SUITE_START: MainThread 23
  0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html
 ...

Testing with stock AVD images (Android 4.x+ only)

Testing with the above configurations is ideal, but in some cases you may prefer to use stock AVD device images from the Nexus profiles available.

1. From the Android Virtual Device (AVD) Manager, click 'Create....'
2. Choose a Nexus phone or tablet device configuration. (Nexus 9 for tablets, Nexus 5 for phones is recommended).
3. Choose a system image with API level 15 or 18. (If you're not sure if you want arm or x86 then you probably want arm). If no suitable image is available, one can be installed with the Android SDK Manager.
4. Next, check 'Use Host GPU' if possible. See Using Hardware Acceleration.
5. Hit Finish and that should be it. If you have setup Firefox for Android to run on your IDE, you should be able to launch the emulator straight from there as well.

Multiple emulators

It's also possible to test with multiple emulators by correctly setting the ANDROID_SERIAL environment variable to the device ID seen in adb devices:

   $ adb devices
   List of devices attached
   emulator-5554	device
   emulator-5556	device

To make Robocop (for example) run on emulator-5556:

   $ export ANDROID_SERIAL=emulator-5556
   $ mach robocop # Runs your tests as normal

Device Managers

Most test suites - mochitests, reftests, xpcshell tests, and others - use a "device manager" module to communicate with the remote device. There are two device manager implementations: ADB and SUT.

The ADB device manager uses the adb command from the Android SDK to communicate with the remote device. To use the ADB device manager:

• ensure the adb command is in your shell's PATH
• mach test commands use adb by default; if running tests with make, set environment variable DM_TRANS=adb

The SUT device manager uses TCP to communicate with a remote agent, which must be installed on the device. To use the SUT device manager:

• ensure TCP connectivity between the local host and the remote device: check that they are on the same network and you can ping each from the other
• ensure the SUT agent is installed and started on the remote device
  • the SUT agent APK is built alongside Firefox for Android; just install <objdir-droid>/build/mobile/sutagent/android/sutAgentAndroid.apk
  • The agent should be configured to start automatically with your phone when it boots. To start it immediately from an adb shell execute: `am start -n com.mozilla.SUTAgentAndroid/.SUTAgentAndroid -a android.intent.action.MAIN`
• if running tests with make, set environment variable DM_TRANS=sut and set TEST_DEVICE=<ip address of device -- displayed by SUT agent>

Host Builds (MOZ_HOST_BIN)

Android mochitests and reftests are driven by test suites on a host machine running xpcshell. The Android device being driven is referred to as the target device. The test suite locates xpcshell on the host machine via the environment variable MOZ_HOST_BIN, which must point to the directory that contains the xpcshell binary (executable on the host machine), its associated executables (certutil, pk12util, ssltunnel, etc), and its shared libraries.

When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you.

Quick setup

You can download prebuilt host utilities:

Host architecture	Size	Download link
Mac OS X	76MB	http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
Linux (64-bit)	46MB	http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-x86_64.tar.xz
Linux (32-bit)	46MB	http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-i686.tar.xz

Extract the archives using tar (or tar and xzip), like:

 cd ~/.mozbuild
 wget http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
 tar Jxf host-utils-37.0a2.en-US.mac.tar.xz
 export MOZ_HOST_BIN=~/.mozbuild/host-utils.37.0a2.en-US.mac

Packaging the host utility archives listed above is easy: follow these instructions.

Advanced setup

Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the getxre utility that comes with JimDB. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by getxre. (getxre does what the packaging instructions describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace $DIR with an output directory, such as ~/.mozbuild/host-utils.

 wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py
 python getxre.py -d $DIR
 export MOZ_HOST_BIN=$DIR/bin

Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:

 ac_add_options --enable-application=browser
 mk_add_options MOZ_OBJDIR=./objdir-desktop

Then execute (note that MOZ_HOST_BIN must specify an absolute path):

 MOZCONFIG=mozconfig.desktop ./mach build
 export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin

On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":

 LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.

Test Directory

All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).

The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:

• the device may not contain an SD card, or the SD card may not be mounted
• there may not be enough free space on the default location's partition
• the default location may not be writable by the ADB shell and/or SUT agent

(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)

If necessary, the default remote test directory may be changed with:

 --remoteTestRoot=<remote-directory>

talos

• NOTE: Talos will be retired and replaced by new tests on autophone before the end of 2015.
• NOTE: this requires a fix for bug 1122701
• NOTE: this requires python 2.7

See also https://wiki.mozilla.org/Buildbot/Talos

 hg clone http://hg.mozilla.org/build/talos talos
 cd talos
 python INSTALL.py
 . bin/activate
 cd talos
 <obtain your fennec apk and put it in the current directory>
 <obtain your robocop apk and put it in the current directory>
 <obtain your fennec_ids.txt and put it in the current directory>
 <install your fennec apk>
 <if running a robocop-based test, install robocop.apk>
 <start sutagent on the device>
 python remotePerfConfigurator.py --apkPath=fennec-38.0a1.en-US.android-arm.apk 
    -v -e org.mozilla.fennec 
    --webServer=<your local desktop IP>:8080 
    --noChrome 
    --remoteDevice=<sutagent-IP> 
    --sampleConfig=remote.config --output=local.yml --browserWait=60 
    --activeTests=tcheck2 
    --fennecIDs=fennec_ids.txt
 python run_tests.py --apkPath=fennec-38.0a1.en-US.android-arm.apk --noisy local.yml

For Robocop based tests (tcheck2, tprovider, etc...), we need to use the --fennecIDs flag to pass in the generated fennec_ids.txt file from the build you are testing. This file is generated during build time and has to match the fennec.apk and robocop.apk file. When this flag is used, we copy fennec_ids.txt and robotium.config (generated during configuration time) to the device and use those to run Robocop. If you are running ts, tp4, tsvg, or other traditional talos tests, there is no need for the --fennecIDs flag.

Aside: For a quick-and-dirty hacky way to run robocop-talos tests locally, see Mobile/Fennec/Android/LocalRoboTalos

S1/S2 Automation

These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times.

S1/S2 graphs can be viewed at: http://phonedash.mozilla.org/

Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2

See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests

Eideticker

Eideticker measures perceived Firefox performance by video capturing automated browser interactions.

Eideticker is no longer running Android tests. Historical data can be viewed at: http://eideticker.mozilla.org/

See also:

• • http://wrla.ch/blog/2011/11/measuring-what-the-user-sees/
  • http://wrla.ch/blog/2012/03/announcing-the-eideticker-mobile-performance-dashboard/
  • https://github.com/mozilla/eideticker

Trouble-shooting testing problems

• Does your mozconfig contain "ac_add_options --disable-tests"?
  • If so, you will see something like:

make: *** No rule to make target <your-test-target>.  Stop.

• Is adb in your $PATH?
• Is your device connected? Does it appear in the output from "adb devices"?
• Can you run adb shell?