Mobile/Fennec/Android/Testing: Difference between revisions

From MozillaWiki
< Mobile‎ | Fennec‎ | Android
Jump to navigation Jump to search
Line 404: Line 404:


See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests
See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests
== Eideticker ==
Eideticker measures perceived Firefox performance by video capturing automated browser interactions.
Eideticker is no longer running Android tests. Historical data can be viewed at: http://eideticker.mozilla.org/
See also:
** http://wrla.ch/blog/2011/11/measuring-what-the-user-sees/
** http://wrla.ch/blog/2012/03/announcing-the-eideticker-mobile-performance-dashboard/
** https://github.com/mozilla/eideticker


== Trouble-shooting testing problems ==
== Trouble-shooting testing problems ==

Revision as of 16:48, 1 August 2018

This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.

Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.

For the impatient...

mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android: 

 mach robocop
 mach mochitest
 mach reftest
 mach crashtest
 mach jstestbrowser
 mach xpcshell-test
 mach cppunittest
 mach geckoview-junit

They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.

Quick reference

As a front-end dev, the following tests will be regularly useful:

Name Results name (TH) Auto? Local command Description More
Robocop rc* N ./mach robocop On-device UI tests link
JUnit4 tests test (tier 2) Y ./mach gradle app:test [0] Java unit test suite link
integration N/A N/A via IDE On-device integration tests link

As well as the following static analysis tools:

Name Results name (TH) Auto? Local command Description More
Checkstyle checkstyle (tier 2) Y ./mach gradle app:checkstyle Reports style violation in Java code link
Android Lint lint (tier 2) Y ./mach gradle app:lint Detects common errors in Android code & resources link
eslint ES (lint opt: tier 2) Y ./mach eslint mobile/android [1] Checks for JS errors (e.g. syntax errors) link

Key:

  • TH stands for Treeherder
  • Auto? refers to jobs that run automatically on Treeherder when the associated files change. For more info, see the docs on automatic tasks.
  • More links to more information regarding a test

[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See #JUnit4 tests.
[1]: You must first run a setup command – see #eslint

Test Environment

When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:

  • a "host" computer, running Linux or OSX
  • a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer
  • a Firefox for Android build, including an apk
  • "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform
  • a "device manager" to communicate with the Android device
  • a TCP/IP network connection between host and device

A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.

Running tests with mach simplifies environment concerns significantly:

  • If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.
  • If Firefox for Android is not installed on the device, mach will offer to install Firefox.
  • If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.

Front-end-centric

(In the interest of removing duplication) For basic details & run instructions, see #Quick reference.

JUnit4 tests

  • Runs on Robolectric, which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device
  • Supports Mockito for custom mocking (see TestVerifyAction for a sample).
  • Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.


Troubleshooting

integration tests

  • Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.
  • There is no way to run these tests from the command line
  • These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.

robocop UI tests

General Robocop information and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.

The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.

To run robocop tests, first build and install Firefox for Android, 

 mach build
 mach package
 mach install

Next, execute mach robocop which installs the Robocop APK to the device and starts testing the entire test suite. 

 mach robocop

or 

 mach robocop <test-name>

If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall. 

 mach build mobile/android/tests/browser/robocop
 mach robocop

This builds the tests in mobile/android/tests/browser/robocop and then installs the debug-signed Robocop APK onto the device. (Building mobile/android also builds the tests within mobile/android/tests.)

Notes:

  • Mach is self documenting! For help, try mach help robocop.
  • To run one test at a time, find the test name (like "testLoad") in mobile/android/tests/browser/robocop/robocop.ini and pass it as an argument, like: mach robocop testLoad.
  • A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.
  • Additional tips at Auto-tools/Projects/Robocop#Frequently_found_errors

Static analysis

There are some other tools to be found at the Not yet in common use page.

checkstyle

Android Lint

eslint

To use eslint, you must first set it up: 

./mach eslint --setup  # run once, or if the command breaks for some reason
./mach eslint mobile/android

Other automation tasks

android-api-15-gradle-dependencies

This job is used to get our gradle and application dependencies in a format usable by the builders. See readthedocs for more info.

mochitest (plain and chrome)

General mochitest info General mochitest-chrome info

Pre-requisites:

  • Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install
  • Ensure your device is connected and visible with "adb devices".
  • Ensure that the device and host machine are on the same network <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is not sufficient.

Running tests: 

 mach mochitest --flavor plain|chrome
 OR mach mochitest <test-dir>
 OR mach mochitest <test-dir>/<test-name>

Use "find -name mochitest.ini" to find valid test directories for mochitest plain.

Use "find -name chrome.ini" to find valid test directories for mochitest chrome.

xpcshell

General xpcshell test information.

Pre-requisites:

  • Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.
  • Ensure your device is connected and visible with "adb devices".

To run all tests referenced by the master xpcshell manifest: 

 mach xpcshell-test

To run a subset of tests in the specified directory: 

 mach xpcshell-test <test-directory>
 OR mach xpcshell-test <test-directory>/<test-name>

Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup: 

 mach xpcshell-test <test-directory> --no-setup
 OR mach xpcshell-test <test-directory>/<test-name> --no-setup

Notes:

  • A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.
  • Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.

cppunittests

General cppunit test information

Pre-requisites:

  • Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.
  • Ensure your device is connected and visible with "adb devices".

To run a single compiled code test: 

 mach cppunittest <test>

For example, 

 mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers

To run all the compiled code tests listed in the master cppunittests.ini manifest: 

 mach cppunittest

Notes:

  • Specifying the test by relative path is difficult; specify an absolute path to the test binary.
  • It is not possible to run all tests in a directory.
  • All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.

reftests (and crashtests and js-reftests)

General reftest information.

Pre-requisites:

  • Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install
  • Ensure your device is connected and visible with "adb devices".
  • Ensure that the device and host machine are on the same network <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is not sufficient.

Running reftests: 

 mach reftest
 OR mach reftest <test-dir>
 OR mach reftest <test-dir>/<test-name>

Use "find -name reftest.list" to find valid test directories.

Running crashtests: 

 mach crashtest
 OR mach crashtest <test-dir>
 OR mach crashtest <test-dir>/<test-name>

Use "find -name crashtest.list" to find valid test directories.

Running js-reftests: 

 mach jstestbrowser

Notes:

  • There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).

Running tests on the Android emulator

The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 4.2 x86 opt" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: Running the same tests on the same emulator on different host hardware may produce different results.

Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple: 

 ./mach android-emulator --version 4.3

That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)

To use the Android 4.2 x86 image: 

 ./mach android-emulator --version x86

(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is much faster than the arm emulator.)

The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be much faster.

If you want to "reset" an image (throw away any installed apks and/or settings changes): 

 ./mach android-emulator --force-update

will discard the previous image and download a new one.

Once an emulator is running, you can run tests against it just like any other Android device. For example: 

 ./mach android-emulator && ./mach install && ./mach mochitest

For the very lazy, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get: 

 $ ./mach mochitest testing/mochitest/tests/Harness_sanity
 No Android devices connected. Start an emulator? (Y/n) y
 Starting emulator running Android 4.3...
 It looks like Firefox is not installed on this device.
 Install Firefox? (Y/n) y
 Installing Firefox. This may take a while...
 From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.
 ######
 ### Now running mochitest-plain.
 ######
  0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests
  0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...
  0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...
  0:01.82 SUITE_START: MainThread 23
  0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html
 ...

Testing with stock AVD images (Android 4.x+ only)

Testing with the above configurations is ideal, but in some cases you may prefer to use stock AVD device images from the Nexus profiles available.

  1. From the Android Virtual Device (AVD) Manager, click 'Create....'
  2. Choose a Nexus phone or tablet device configuration. (Nexus 9 for tablets, Nexus 5 for phones is recommended).
  3. Choose a system image with API level 16 or 18. (If you're not sure if you want arm or x86 then you probably want arm). If no suitable image is available, one can be installed with the Android SDK Manager.
  4. Next, check 'Use Host GPU' if possible. See Using Hardware Acceleration.
  5. Hit Finish and that should be it. If you have setup Firefox for Android to run on your IDE, you should be able to launch the emulator straight from there as well.

Multiple emulators

It's also possible to test with multiple emulators by correctly setting the ANDROID_SERIAL environment variable to the device ID seen in adb devices: 

   $ adb devices
   List of devices attached
   emulator-5554	device
   emulator-5556	device

To make Robocop (for example) run on emulator-5556: 

   $ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal

Host Builds (MOZ_HOST_BIN)

Android mochitests and reftests are driven by test suites on a host machine running xpcshell. The Android device being driven is referred to as the target device. The test suite locates xpcshell on the host machine via the environment variable MOZ_HOST_BIN, which must point to the directory that contains the xpcshell binary (executable on the host machine), its associated executables (certutil, pk12util, ssltunnel, etc), and its shared libraries.

When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it. If a change to the host utilities is required, follow these instructions.

Advanced setup

Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the getxre utility that comes with JimDB. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by getxre. (getxre does what the packaging instructions describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace $DIR with an output directory, such as ~/.mozbuild/host-utils. 

 wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py
 python getxre.py -d $DIR
 export MOZ_HOST_BIN=$DIR/bin

Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as: 

 ac_add_options --enable-application=browser
 mk_add_options MOZ_OBJDIR=./objdir-desktop

Then execute (note that MOZ_HOST_BIN must specify an absolute path): 

 MOZCONFIG=mozconfig.desktop ./mach build
 export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin

On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".": 

 LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.

Test Directory

All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).

The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:

  • the device may not contain an SD card, or the SD card may not be mounted
  • there may not be enough free space on the default location's partition
  • the default location may not be writable by the ADB shell and/or SUT agent

(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)

If necessary, the default remote test directory may be changed with: 

 --remoteTestRoot=<remote-directory>

S1/S2 Automation

These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times.

S1/S2 graphs can be viewed at: http://phonedash.mozilla.org/

Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2

See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests

Trouble-shooting testing problems

  • Does your mozconfig contain "ac_add_options --disable-tests"?
  • Is adb in your $PATH?
  • Is your device connected? Does it appear in the output from "adb devices"?
  • Can you run adb shell?
  • Ensure the device's screen is on.
  • Ensure that the device and host machine are on the same network.
    • Can you ping the device from the host? Can you ping the host from the device?
    • Are the phone and the desktop both using wifi? (wifi vs ethernet??)
    • Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)
    • Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.
  • Ensure the test harness has the correct IP address for your machine
    • Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.
  • If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
Retrieved from "https://wiki.allizom.org/index.php?title=Mobile/Fennec/Android/Testing&oldid=1198335"