Mobile/Fennec Unittests/Remote Testing

From MozillaWiki
Jump to navigation Jump to search

Overview

While battling Windows Mobile and python, [we came to the conclusion] that we needed to shift out of our model where we run everything on the device and allow for some or all test tools and data to be run remotely.

This has proven to be a reliable model and will meet our needs with Android. We also have found this very useful for investigating tests on Maemo as we don't need to ssh to the device and do all the setup.

Requirements / Setup

Testing remote requires a few things:

  • a platform that you can install an agent on (currently, Windows Mobile (c++), Android (java), and Maemo/Linux (python). NOTE: a generic C++ linux agent is in development
  • a recent build or downloaded tests package (from mozilla-central, not 1.9.2)
  • the ip address of your device
  • python on your host machine to run the test harness runners

To set this up, you need to first get the test agent for your platform here. Copy that to your device and set it up to run automatically on boot.

  • NOTE: fully implemented agents have a GUI and will give you useful information like ip address, commands running, and audible alerts

Next you need to determine your ip address. This can be done with a fully implemented agent, command line tools, or looking at your router.

Once this is running lets test it from your desktop (host):

  • telnet <ip> 20701

Now that you have verified you can connect, the next step is to start writing python code. We have a library (devicemanager.py) which allows you to do common (file and process) operations across the telnet interface.

Assumptions

Since we developed this for Mozilla Fennec and Firefox on remote devices, we have made some assumptions which you should be aware of.

  • / for all platforms, even windows
    • Our agents will convert / to \ if the device uses \. This can cause confusion in commands with arguments such as a browser with http://url. We special case the :// type arguments, so keep that in mind.
  • /tests is available as the root of everything
    • To simplify things, we assume that /tests is a valid path. If it isn't, we special case that in the agent (as seen in the python agent). In our automation scripts, we rely on /tests for log file paths, profile directories, application, and data.
  • install application to /tests/appname
    • The existing automation scripts depend on your application (Fennec/Firefox) to be installed in /tests/<appname> (ex. /tests/fennec.) This decision helped simplify things. This is done by pushing and unzipping the application package file to the /tests directory.
  • default telnet port is 20701, data channel is 20700.
    • in the past we started with 20720, but as we evolved, we have standardized on 20701. We have a data channel defined on port 20700 for the heartbeat and output of some commands (like exec and unzp.) This is important since our integration with buildbot depends on this for monitoring device state.

Available Agent Commands

run [executable] [args]  - start program no wait
exec [executable] [args] - start program wait
fire [executable] [args] - start program no wait
arun [executable] [args] - start program no wait
kill [program name]      - kill program no path
ps                       - list of running processes
info                     - list of device info
     [os]                - os version for device
     [id]                - unique identifier for device
     [uptime]            - uptime for device
     [systime]           - current system time on device
     [screen]            - width, height and bits per pixel for device
     [memory]            - physical, free, available, storage memory for device
     [processes]         - list of running processes see 'ps'
alrt [on/off]            - start or stop sysalert behavior
disk [arg]               - prints disk space info
cp file1 file2           - copy file1 to file2 on device
time file                - timestamp for file on device
hash file                - generate hash for file on device
cd directory             - change cwd on device
cat file                 - cat file on device
cwd                      - display cwd on device
mv file1 file2           - move file1 to file2 on device
push filename            - push file to device
rm file                  - delete file on device
rmdr directory           - delete empty directory on device
prune directory          - delete directory on device even if not empty
mkdr directory           - create directory on device
dirw directory           - tests whether the directory is writable on the device
stat processid           - stat process on device
dead processid           - print whether the process is alive or hung on device
mems                     - dump memory stats on device
ls                       - print directory on device
tmpd                     - print temp directory on device
ping [hostname/ipaddr]   - ping a network device
rebt                     - reboot device
unzp archivename [dir]   - unzip a zipfile respecting directories optionally into dir
clok                     - print current device time
quit                     - reset all network connections on SUTAgent
exit                     - shutdown the SUTAgent
help                     - you're reading it

DeviceManager library

sendCMD(cmdline, newline = True, sleep = 0)
stripPrompt(data)
pushFile(localname, destname)
mkDir(name)
mkDirs(filename)
pushDir(localDir, remoteDir)
dirExists(dirname)
fileExists(filepath)
listFiles(rootdir)
removeFile(filename)
removeDir(remoteDir)
getProcessList()
getMemInfo()
fireProcess(appname)
launchProcess(cmd, outputFile = "process.txt", cwd = '')
communicate(process, timeout = 600)
poll(process)
processExist(appname)
killProcess(appname)
getTempDir()
getFile(remoteFile, localFile = '')
getDirectory(remoteDir, localDir)
validateFile(remoteFile, localFile)
getRemoteHash(filename)
getLocalHash(filename)
getDeviceRoot()
getAppRoot()
getTestRoot(type)
signal(processID, signalType, signalAction)
getReturnCode(processID)
unpackFile(filename)
reboot(wait = False)
validateDir(localDir, remoteDir)

Mozilla Unittests

We have successfully run all our automation tests on a remote device. Here are links to the code and so notes about how these are run for anybody extending or debugging.

First off, we did a lot of work to refactor all the existing python harnesses (talos, mochitest, reftest, xpcshell) into classes and smaller methods. This was necessary in order to subclass these scripts so we can overload certain methods and add or remove specific functionality.

In addition to the refactoring, we had to change a lot of the tests to remove the hardcoded localhost and 127.0.0.1 references. This was a large change, but it gave us the option to run all mochitest (plain not chrome), talos, and reftest tests from a remote webserver. This is the recommended way to run all the tests to avoid setting up a webserver on the device.

As a developer working on testing your code or fixing a remote test, please take note that there could be a small issue with a test running remotely vs a real failure on the product.

Mochitest ([runtestsremote.py]):

  • this is a subclass of automation.py and runtests.py.
  • Overloads the Process class defined in automation.py to use devicemanager.py
  • adds extra command line arguments
    • --deviceIP, --devicePort, --remote-logfile, --remote-webserver, --http-port, --ssl-port, --remoteProductName
    • --remote-logfile requires knowledge of the remote system and the full path to the log file, not /tests. This is embedded in the url and we are not going to be munging that
  • assumes fennec is installed in /tests/fennec
  • assumes profile in /tests/mochitest

Reftest([remotereftest.py]):

  • subclass of automation.py and reftest.py
  • Overloads the Process class defined in automation.py to use devicemanager.py
  • adds extra command line arguments
    • --device, --devicePort, --remoteProductName, --remoteAppName, --remote-webserver
  • assumes fennec is installed in /tests/fennes
  • assumes reftest.jar is included in the fennec build, not copied to it as we do on desktop
  • assumes profile in /tests/reftest
  • currently doesn't work with http(...) reftests

Talos([code]):

  • retrofitted PerfConfigurator.py and run_tests.py scripts to use devicemanager.py library for os and process calls
  • added a winmo platform type for process, setup, and profile management
  • profile needs to allow xpconnect for remote host, not localhost
  • tested in production for ts, but still need to verify tsvg, tp4, tpan, tzoom

Xpcshell([remotexpcshelltests.py]):

  • subclass of runxpcshelltests.py
  • adds extra command line arguments
    • --device
  • assumes fennec is installed in /tests/fennes
  • assumes profile in /tests/xpcshell
  • will create a .zip of the xpcshell/* tests and copy that to device under /tests/xpcshell

DeviceManager Examples

Copy File to device: 

import devicemanager
dm = devicemanager.DeviceManager('192.168.1.103', 20701)
dm.pushFile('/home/joel/config.txt', '/config.txt')

Remote Testing Examples

There as some examples of running remote tests.


from a build environment

This requires a host build to run xpcshell + httpd.js as a webserver (for all but talos). What I normally do is install a build of Fennec on my Android board, then from a desktop build environment (note these have to be the same version so the extensions installed from the desktop build will run on the device), I do this:

  • mochitest
cd $(objdir)/_tests/testing/mochitest
python runtestsremote.py --deviceIP=<your.device.ip.addr> --xre-path=../../../dist/bin --app=org.mozilla.fennec --test-path=gfx
# add a '--dm_trans=adb' if you are connected to your device locally and don't have a SUT agent installed
  • reftest
cd $(objdir)/_tests/reftest
python remotereftest.py --deviceIP=<your.device.ip.addr> --xre-path=../../../dist/bin --app=org.mozilla.fennec tests/layout/reftest/reftest-sanity/reftest.list
# add a '--dm_trans=adb' if you are connected to your device locally and don't have a SUT agent installed
  • talos
hg clone http://hg.mozilla.org/build/talos
#configure apache to point the web root at the talos directory (http://localhost/getInfo.html should resolve)
cd talos
python remotePerfConfigurator.py -e org.mozilla.fennec --remoteDevice <your.device.ip.addr> --webServer <your.apache.ip.addr> --resultsServer ' ' --resultsLink ' ' --activeTests tpan --output tpan.yml
python run_tests.py -d -n tpan.yml
# NOTE: we don't have --dm_trans=adb support in talos yet
Retrieved from "https://wiki.allizom.org/index.php?title=Mobile/Fennec_Unittests/Remote_Testing&oldid=314103"