Auto-tools/Projects/Robocop

Problem

With advent of NativeFennec, a new framework was needed to test the UI of the Mobile Browser. Robotium was identified as a useful framework to take advantage of, but is not complete. Robocop incorporates all of Activity, Instrumentation, and Robotium Classes to inject events into the new Java front-end of Fennec.

Setup

Step one: Build Native Fennec

First grab the source
Then setup your build environment
Finally, build with the command
```
make -f client.mk
```
in the source directory

Step two: Build Firefox (Necessary for XPCShell)

Follow these instructions; the source used here, although the same as Fennec's, should live in a separate folder

Step three: Install Native Fennec

Run the following commands in sequence, from the Fennec source directory:

cd {objdir}
make package
adb install dist/fennec-*.apk

NOTE: {objdir} is a placeholder; the name of your object directory should not matter.
NOTE 2: Fennec cannot be built on Windows.

Running tests

Set up the MOZ_HOST_BIN environment variable

The MOZ_HOST_BIN variable will need to be set to the directory with XPCShell on your machine. You can execute the following command in the terminal, or it can be saved into .bashrc / .bash_profile for convinence:

export MOZ_HOST_BIN=/ABSOLUTE/PATH/TO/FIREFOX/SOURCE/{objdir}/dist/bin

If you're having difficulty finding the absolute path of XPCShell's directory, simply locate XPCShell in your {objdir}/dist/bin and execute the command pwd .

Again, {objdir} is a placeholder.

Running all the tests

From the Fennec's objdir:

make mochitest-robotium

Running a single test

From the Fennec's objdir (using testLoad as an example):

TEST_PATH=testLoad make mochitest-robotium

Updating your test directory

It is not always necessary to rebuild Fennec to use new tests. If you are updating the build with your own test, ensure that your test has been added to the manifest in /PATH/TO/FENNEC/SOURCE/mobile/android/base/tests/robocop.ini .

Otherwise, you can retrieve tests from the repository by executing:

hg pull
hg update

Now you can build the robocop part of Fennec by executing:

cd objdir
make -C build/mobile/robocop/
make package

Frequently found Errors

OOM Error. This is an error found after use of Robocop frequently, but once it has happened, Fennec itself continues to crash. To fix it, simply uninstall and reinstall. This does not occur of frequent use of Robocop, as I have run 30 iterations of just robocop without it throwing this error.

InvocationTargetException Error. This is an issue arising when using a Tegra.It looks like the following:

W/System.err( 2610): java.lang.reflect.InvocationTargetException
W/System.err( 2610): at org.mozilla.gecko.GeckoAppShell.unregisterGeckoEventListener(GeckoAppShell.java:1565)
...

Build Error. This is resolved by doing a clean build.

make[7]: Entering directory `/media/sdan/birch/objdir-native-droid/mobile/android/base/locales'
make[7]: Leaving directory `/media/sdan/birch/objdir-native-droid/mobile/android/base/locales'
res/values/strings.xml:1: error: Error parsing XML: no element found
res/layout/awesomebar_search.xml:9: error: Error: No resource found that matches the given name (at 'hint' with value '@string/awesomebar_default_text').
res/layout-v11/awesomebar_search.xml:9: error: Error: No resource found that matches the given name (at 'hint' with value '@string/awesomebar_default_text').
res/layout/browser_toolbar.xml:7: error: Error: No resource found that matches the given name (at 'hint' with value '@string/awesomebar_default_text').
res/layout/crash_reporter.xml:6: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_message').
res/layout/crash_reporter.xml:14: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_help_message').
res/layout/crash_reporter.xml:21: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_send_report_message').
...

Infrequent Runtime Errors. Two have been known to occur: One, where Fennec freezes when loading a page, the other where the page loads, but has 0 content. These are worked around by restarting the test.

What happens when a test is run

Our tests are run from the talos and mochitest scripts. These rely on devicemanager and other scripts/tools in the harness directory. Here is what happens:

create a [fresh profile] and copy it to the device (/mnt/sdcard/tests/profile)
we generate robotium.config and push it to the device

$ adb shell cat /mnt/sdcard/tests/robotium.config
profile=/mnt/sdcard/tests/profile            # from the first step
logfile=/mnt/sdcard/tests/logs/mochitest.log # where robocop will write logs to
host=http://mochi.test:8888/tests            # mochi.test is defined in the profile proxy auto config
rawhost=http://192.168.1.73:8888/tests       # used for some tests

we push fennec_ids.txt to the device (/mnt/sdcard/tests/fennec_ids.txt)
for each test case, in [robocop.ini] we launch fennec. Here is an example of the first test case testallpagesTab:

am instrument -w -e deviceroot /mnt/sdcard/tests -e class org.mozilla.fennec.tests.testAllPagesTab org.mozilla.roboexample.test/org.mozilla.fennec.FennecInstrumentationTestRunner

Notes

If you are writing tests for Robocop, see [Writing Tests] for more API information
We are using [NativeUI Messages] to talk between Robocop and gecko
There has been interest in using a record/replay tool similar to [testdroid]. Most likely this will allow for people to record something and then we can hand edit it into a proper test case.
adb shell am instrument -w org.mozilla.roboexample.test/android.test.InstrumentationTestRunner
Trevor's Presentation on it can be found in [This zip]