Mobile/Fennec/Android/Testing: Difference between revisions

Jump to navigation Jump to search

Mobile/Fennec/Android/Testing (view source)

Revision as of 21:03, 17 September 2015

92 bytes added ,  17 September 2015
no edit summary
No edit summary
Line 12: Line 12:


They will all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.
They will all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.
== Device Managers ==
Most test suites - mochitests, reftests, xpcshell tests, and others - use a "device manager" module to communicate with the remote device. There are two device manager implementations: ADB and SUT.
The ADB device manager uses the adb command from the Android SDK to communicate with the remote device. To use the ADB device manager:
* ensure the adb command is in your shell's PATH
* set environment variable DM_TRANS=adb
* do not set environment variable TEST_DEVICE, or set TEST_DEVICE= (unless you really need it)
The SUT device manager uses TCP to communicate with a remote agent, which must be installed on the device. To use the SUT device manager:
* ensure TCP connectivity between the local host and the remote device: check that they are on the same network and you can ping each from the other
* ensure the SUT agent is installed and started on the remote device
** the SUT agent APK is built alongside Fennec; just install <objdir-droid>/build/mobile/sutagent/android/sutAgentAndroid.apk
** The agent should be configured to start automatically with your phone when it boots. To start it immediately from an adb shell do: `am start -n com.mozilla.SUTAgentAndroid/.SUTAgentAndroid -a android.intent.action.MAIN`
* set environment variable DM_TRANS=sut
* set environment variable TEST_DEVICE=<ip address of remote device -- displayed by SUT agent>
Note: to run the tests you should ensure that your host machine and device are on the same network so that the device can contact the server that will run on the host.
== Host Builds (MOZ_HOST_BIN) ==
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.
=== Quick setup ===
You can download prebuilt host utilities:
{| class="wikitable"
|-
! Host architecture !! Size !! Download link
|-
| Mac OS X || 76MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
|-
| Linux (64-bit) || 46MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-x86_64.tar.xz
|-
| Linux (32-bit) || 46MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-i686.tar.xz
|}
Extract the archives using tar (or tar and xzip), like:
  cd ~/.mozbuild
  wget http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
  tar Jxf host-utils-37.0a2.en-US.mac.tar.xz
  export MOZ_HOST_BIN=~/.mozbuild/host-utils.37.0a2.en-US.mac
Packaging the host utility archives listed above is easy: follow [[Packaging Android host utilities|these instructions]].
=== Advanced setup ===
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.
  wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py
  python getxre.py -d $DIR
  export MOZ_HOST_BIN=$DIR/bin
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:
  ac_add_options --enable-application=browser
  mk_add_options MOZ_OBJDIR=./objdir-desktop
Then execute (note that MOZ_HOST_BIN must specify an absolute path):
  MOZCONFIG=mozconfig.desktop ./mach build
  export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":
  LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.
== Test Directory ==
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:
* the device may not contain an SD card, or the SD card may not be mounted
* there may not be enough free space on the default location's partition
* the default location may not be writable by the ADB shell and/or SUT agent
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device.  When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)
If necessary, the default remote test directory may be changed with:
  export EXTRA_TEST_ARGS=--remoteTestRoot=<remote-directory>
== Reftests  ==
First, ensure that MOZ_HOST_BIN has been set according to the [[Mobile/Fennec/Android#Host_Builds_.28MOZ_HOST_BIN.29|directions above]].
  TEST_PATH=<path> make -C <objdir-droid> reftest-remote
Example TEST_PATH:
  TEST_PATH=layout/reftests/reftest-sanity/reftest.list
Notes:
* If TEST_PATH is not specified, *all* reftests will be attempted: This usually fails/hangs and is not recommended. Specify a TEST_PATH or use the --total-chunks and --this-chunk arguments to reduce the number of tests executed.
* sut recommended. Test results displayed and saved to reftest.log and reftest-remote.log.
* adb works also. Test results displayed and saved to reftest.log (additional diagnostics in reftest-remote.log).
* Ensure that the device and host machine are on the same network.
== mochitest (plain and chrome) ==
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]
Pre-requisites:
* Ensure that MOZ_HOST_BIN has been set according to the [[Mobile/Fennec/Android#Host_Builds_.28MOZ_HOST_BIN.29|directions above]].
* Ensure your device is connected and visible with "adb devices".
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.
* Ensure that Fennec has been built and installed: mach build && mach package && mach install
Running tests:
  mach mochitest -f plain|chrome
  OR mach mochitest -f plain|chrome <test-dir>
  OR mach mochitest -f plain|chrome <test-dir>/<test-name>
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.
Notes:
* This mach command only supports the adb device manager.


== Robocop ==
== Robocop ==
Line 173: Line 54:
* Ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
* Ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
*Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]
*Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]
== mochitest (plain and chrome) ==
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]
Pre-requisites:
* Ensure that MOZ_HOST_BIN has been set according to the [[Mobile/Fennec/Android#Host_Builds_.28MOZ_HOST_BIN.29|directions above]].
* Ensure your device is connected and visible with "adb devices".
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.
* Ensure that Fennec has been built and installed: mach build && mach package && mach install
Running tests:
  mach mochitest -f plain|chrome
  OR mach mochitest -f plain|chrome <test-dir>
  OR mach mochitest -f plain|chrome <test-dir>/<test-name>
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.
Notes:
* This mach command only supports the adb device manager.


== xpcshell  ==
== xpcshell  ==
Line 217: Line 120:
* It is not possible to run all tests in a directory.
* It is not possible to run all tests in a directory.
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.
== Reftests  ==
First, ensure that MOZ_HOST_BIN has been set according to the [[Mobile/Fennec/Android#Host_Builds_.28MOZ_HOST_BIN.29|directions above]].
  TEST_PATH=<path> make -C <objdir-droid> reftest-remote
Example TEST_PATH:
  TEST_PATH=layout/reftests/reftest-sanity/reftest.list
Notes:
* If TEST_PATH is not specified, *all* reftests will be attempted: This usually fails/hangs and is not recommended. Specify a TEST_PATH or use the --total-chunks and --this-chunk arguments to reduce the number of tests executed.
* sut recommended. Test results displayed and saved to reftest.log and reftest-remote.log.
* adb works also. Test results displayed and saved to reftest.log (additional diagnostics in reftest-remote.log).
* Ensure that the device and host machine are on the same network.


== browser-chrome ==
== browser-chrome ==
Line 300: Line 219:
     $ export ANDROID_SERIAL=emulator-5556
     $ export ANDROID_SERIAL=emulator-5556
     $ mach robocop # Runs your tests as normal
     $ mach robocop # Runs your tests as normal
== Device Managers ==
Most test suites - mochitests, reftests, xpcshell tests, and others - use a "device manager" module to communicate with the remote device. There are two device manager implementations: ADB and SUT.
The ADB device manager uses the adb command from the Android SDK to communicate with the remote device. To use the ADB device manager:
* ensure the adb command is in your shell's PATH
* set environment variable DM_TRANS=adb
* do not set environment variable TEST_DEVICE, or set TEST_DEVICE= (unless you really need it)
The SUT device manager uses TCP to communicate with a remote agent, which must be installed on the device. To use the SUT device manager:
* ensure TCP connectivity between the local host and the remote device: check that they are on the same network and you can ping each from the other
* ensure the SUT agent is installed and started on the remote device
** the SUT agent APK is built alongside Fennec; just install <objdir-droid>/build/mobile/sutagent/android/sutAgentAndroid.apk
** The agent should be configured to start automatically with your phone when it boots. To start it immediately from an adb shell do: `am start -n com.mozilla.SUTAgentAndroid/.SUTAgentAndroid -a android.intent.action.MAIN`
* set environment variable DM_TRANS=sut
* set environment variable TEST_DEVICE=<ip address of remote device -- displayed by SUT agent>
Note: to run the tests you should ensure that your host machine and device are on the same network so that the device can contact the server that will run on the host.
== Host Builds (MOZ_HOST_BIN) ==
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.
=== Quick setup ===
You can download prebuilt host utilities:
{| class="wikitable"
|-
! Host architecture !! Size !! Download link
|-
| Mac OS X || 76MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
|-
| Linux (64-bit) || 46MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-x86_64.tar.xz
|-
| Linux (32-bit) || 46MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-i686.tar.xz
|}
Extract the archives using tar (or tar and xzip), like:
  cd ~/.mozbuild
  wget http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
  tar Jxf host-utils-37.0a2.en-US.mac.tar.xz
  export MOZ_HOST_BIN=~/.mozbuild/host-utils.37.0a2.en-US.mac
Packaging the host utility archives listed above is easy: follow [[Packaging Android host utilities|these instructions]].
=== Advanced setup ===
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.
  wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py
  python getxre.py -d $DIR
  export MOZ_HOST_BIN=$DIR/bin
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:
  ac_add_options --enable-application=browser
  mk_add_options MOZ_OBJDIR=./objdir-desktop
Then execute (note that MOZ_HOST_BIN must specify an absolute path):
  MOZCONFIG=mozconfig.desktop ./mach build
  export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":
  LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.
== Test Directory ==
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:
* the device may not contain an SD card, or the SD card may not be mounted
* there may not be enough free space on the default location's partition
* the default location may not be writable by the ADB shell and/or SUT agent
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device.  When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)
If necessary, the default remote test directory may be changed with:
  export EXTRA_TEST_ARGS=--remoteTestRoot=<remote-directory>


== talos ==
== talos ==
 
*NOTE: Talos will be retired and replaced by new tests on autophone before the end of 2015.
*NOTE: this requires a fix for {{bug|1122701}}
*NOTE: this requires a fix for {{bug|1122701}}
*NOTE: this requires python 2.7
*NOTE: this requires python 2.7
Gbrown
Confirmed users
1,759

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1095909"

Navigation menu