View source for Mobile/Fennec/Android/Testing

This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.

Having trouble? Ping :gbrown on #mobile.

== For the impatient... ==
mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:

  mach robocop
  mach mochitest
  mach reftest
  mach crashtest
  mach jstestbrowser
  mach xpcshell-test
  mach cppunittest
  mach geckoview-junit
  mach web-platform-test
  mach marionette-test
  mach test

They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.

=== Quick reference ===
As a front-end dev, the following tests will be regularly useful:
{| class="wikitable sortable"
|-
! Name
! Results name (TH)
! Auto?
! Local command
! Description
! More
|-
| Robocop
| rc*
| N
| <tt>./mach robocop</tt>
| On-device UI tests
| [[#robocop UI tests|link]]
|-
| JUnit4 tests
| test (tier 2)
| Y
| <tt>./mach gradle app:test</tt> [0]
| Java unit test suite
| [[#JUnit4 tests|link]]
|-
| integration
| N/A
| N/A
| via IDE
| On-device integration tests
| [[#integration tests|link]]
|}

As well as the following static analysis tools:
{| class="wikitable sortable"
|-
! Name
! Results name (TH)
! Auto?
! Local command
! Description
! More
|-
| Checkstyle
| checkstyle (tier 2)
| Y
| <tt>./mach gradle app:checkstyle</tt>
| Reports style violation in Java code
| [[#checkstyle|link]]
|-
| Android Lint
| lint (tier 2)
| Y
| <tt>./mach gradle app:lint</tt>
| Detects common errors in Android code & resources
| [[#Android Lint|link]]
|-
| eslint
| ES (lint opt: tier 2)
| Y
| <tt>./mach eslint mobile/android</tt> [1]
| Checks for JS errors (e.g. syntax errors)
| [[#eslint|link]]
|}

Key:
* <b>TH</b> stands for Treeherder
* <b>Auto?</b> refers to jobs that run automatically on Treeherder when the associated files change. For more info, see [[Mobile/Fennec/Testing/Understanding_treeherder#Automatic_tasks|the docs on automatic tasks]].
* <b>More</b> links to more information regarding a test

[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See [[#JUnit4 tests]].<br>
[1]: You must first run a setup command – see [[#eslint]]

== Test Environment ==
When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:
* a "host" computer, running Linux or OSX
* a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer
* a Firefox for Android build, including an apk
* "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform
* a "device manager" to communicate with the Android device
* a TCP/IP network connection between host and device

A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.

Running tests with mach simplifies environment concerns significantly:
* If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.
* If Firefox for Android is not installed on the device, mach will offer to install Firefox.
* If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.

== Front-end-centric ==
(In the interest of removing duplication) For basic details & run instructions, see [[#Quick reference]].

=== JUnit4 tests ===
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.


'''Troubleshooting'''
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).

=== integration tests ===
* Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.
* There is no way to run these tests from the command line
* These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.

=== robocop UI tests ===
[[Auto-tools/Projects/Robocop|General Robocop information]] and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.

The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator.  It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.

To run robocop tests, first build and install Firefox for Android, 

  mach build
  mach package
  mach install

Next, execute <tt>mach robocop</tt> which installs the Robocop APK to the device and starts testing the entire test suite. 

  mach robocop

or

  mach robocop <test-name>

Notes:
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.
* Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]

=== Static analysis ===
There are some other tools to be found at the [[Mobile/Fennec/Android/Testing/Not_yet_in_common_use|Not yet in common use page]].

==== checkstyle ====
* Run it in Intellij/Android Studio with [[Mobile/Fennec/Android/Static_analysis/checkstyle-idea|checkstyle-idea]].
* '''example checks?''' [http://checkstyle.sourceforge.net/google_style.html Google's style guide]
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/checkstyle.xml mobile/android/app/checkstyle.xml]
* '''open issues?''' Issues we care about are blocking the meta
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]

==== Android Lint ====
* '''example checks?''' [https://sites.google.com/a/android.com/tools/tips/lint-checks `lint --show`]
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml mobile/android/app/lint.xml]
* '''open issues?''' Run locally for a complete list.
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]
* If you fix all issues for a warning, modify [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml lint.xml] to change your warning into an error!

==== eslint ====
To use eslint, you must first set it up:
 ./mach eslint --setup  # run once, or if the command breaks for some reason
 ./mach eslint mobile/android

* '''example checks?''' http://eslint.org/docs/rules/
* '''config?''' [http://mxr.mozilla.org/mozilla-central/find?string=.eslint&tree=mozilla-central&hint=mobile%2F mobile/*.eslintrc] & [http://mxr.mozilla.org/mozilla-central/source/.eslintignore root .eslintignore]
* '''open issues?''' Open a .eslintrc and enable checks.
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=939329 meta bug]
* If you fix all issues for a warning, modify the appropriate .eslintrc to change your warning into an error!

=== Other automation tasks ===
==== android-api-15-gradle-dependencies ====
This job is used to get our gradle and application dependencies in a format usable by the builders. See [https://gecko.readthedocs.io/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle readthedocs] for more info.

== mochitest (plain and chrome) ==
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]

Pre-requisites:
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install
* Ensure your device is connected and visible with "adb devices".
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.

Running tests:
  mach mochitest --flavor plain|chrome
  OR mach mochitest <test-dir>
  OR mach mochitest <test-dir>/<test-name>

Use "find -name mochitest.ini" to find valid test directories for mochitest plain.

Use "find -name chrome.ini" to find valid test directories for mochitest chrome.

== xpcshell  ==
[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests General xpcshell test information.]

Pre-requisites:
* Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.
* Ensure your device is connected and visible with "adb devices".

To run all tests referenced by the master xpcshell manifest:

  mach xpcshell-test

To run a subset of tests in the specified directory:

  mach xpcshell-test <test-directory>
  OR mach xpcshell-test <test-directory>/<test-name>

Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:

  mach xpcshell-test <test-directory> --no-setup
  OR mach xpcshell-test <test-directory>/<test-name> --no-setup

Notes:
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.

== cppunittests ==
[https://developer.mozilla.org/en/docs/Compiled-code_automated_tests General cppunit test information]

Pre-requisites:
* Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.
* Ensure your device is connected and visible with "adb devices".

To run a single compiled code test:

  mach cppunittest <test>

For example,

  mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers

To run all the compiled code tests listed in the master cppunittests.ini manifest:

  mach cppunittest

Notes:
* Specifying the test by relative path is difficult; specify an absolute path to the test binary.
* It is not possible to run all tests in a directory.
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.

== reftests (and crashtests and js-reftests) ==
[https://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests General reftest information.]

Pre-requisites:
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install
* Ensure your device is connected and visible with "adb devices".
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.

Running reftests:
  mach reftest
  OR mach reftest <test-dir>
  OR mach reftest <test-dir>/<test-name>

Use "find -name reftest.list" to find valid test directories.

Running crashtests:
  mach crashtest
  OR mach crashtest <test-dir>
  OR mach crashtest <test-dir>/<test-name>

Use "find -name crashtest.list" to find valid test directories.
 
Running js-reftests:
  mach jstestbrowser
 
Notes:

* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).

== Running tests on the Android emulator ==

The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 7.0 x86_64 opt/debug" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.

Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:

  ./mach android-emulator --version 4.3

That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)

To use the Android 7.0 x86_64 image:

  ./mach android-emulator --version x86-7.0

(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is '''much''' faster than the arm emulator.)

The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.

If you want to "reset" an image (throw away any installed apks and/or settings changes):

  ./mach android-emulator --force-update

will discard the previous image and download a new one.

Once an emulator is running, you can run tests against it just like any other Android device. For example:

  ./mach android-emulator && ./mach install && ./mach mochitest

For your convenience, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:

  $ ./mach mochitest testing/mochitest/tests/Harness_sanity
  No Android devices connected. Start an emulator? (Y/n) y
  Starting emulator running Android 4.3...
  It looks like Firefox is not installed on this device.
  Install Firefox? (Y/n) y
  Installing Firefox. This may take a while...
  From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.
  ######
  ### Now running mochitest-plain.
  ######
   0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests
   0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...
   0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...
   0:01.82 SUITE_START: MainThread 23
   0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html
  ...

=== Multiple emulators/devices ===

It is possible to test with multiple emulators or devices using the --deviceSerial argument:
    $ adb devices
    List of devices attached
    emulator-5554	device
    emulator-5556	device

To make Robocop (for example) run on ''emulator-5556'':
    $ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal

== Host Builds (MOZ_HOST_BIN) ==

Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.

'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.
'''
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].

=== Advanced setup ===

Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:

  ac_add_options --enable-application=browser
  mk_add_options MOZ_OBJDIR=./objdir-desktop

Then execute (note that MOZ_HOST_BIN must specify an absolute path):

  MOZCONFIG=mozconfig.desktop ./mach build
  export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin

On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":

  LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.

== Test Directory ==
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).

The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:
* the device may not contain an SD card, or the SD card may not be mounted
* there may not be enough free space on the default location's partition
* the default location may not be writable by the ADB shell and/or SUT agent

(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device.  When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)

If necessary, the default remote test directory may be changed with:

  --remoteTestRoot=<remote-directory>

== Trouble-shooting testing problems ==

* Does your mozconfig contain "ac_add_options --disable-tests"?
* Is adb in your $PATH?
* Is your device connected? Does it appear in the output from "adb devices"?
* Can you run adb shell?
* Ensure the device's screen is on.
* Ensure that the device and host machine are on the same network.
** Can you ping the device from the host? Can you ping the host from the device?
** Are the phone and the desktop both using wifi? (wifi vs ethernet??)
** Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)
** Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.
* Ensure the test harness has the correct IP address for your machine
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
* Is your device rooted? Many test suites require root permissions. Don't want to root your device? Consider using an emulator with 'mach android-emulator'.