Mobile/Fennec Automation

Fennec Automation

In developing Fennec, we need to leverage all the work done for Firefox. As an initial step, we will be porting all the automation that buildbot runs on Firefox and we will run it on Fennec.

There are a lot of new variables as we have non standard devices to test on, tests and tools which don't exist or are invalid in the mobile environment and code which needs to be updated to support a new product and environment.

Below is a table outlining what is needed and where we are. Below the table is a log of actions which is intended for people to see what has been done and when if they want more details about a specific item.

Maemkit Toolset

NEW: Download the initial drop of [maemkit-chunked].

Until we get a lot of changes backported into the build system (makefiles, test running scripts, etc...) we will have a temporary test framework that wraps around the build system and test scripts to run all tests on maemo. We will call this the maemkit.

Here are a list of things the maemkit needs to do:

• Package the tests
  • mochitest: tar -chf $(xr_objdir)/_tests/ $(objdir)/_profile/ tests.tar
  • chrometest: same as mochitest
  • reftest: run reftest_extract.py
  • crashtests: run crashtest_extract.py
• copy to device/unpack (leave manual for now)
• run tests in a chunked method
  • mochitest - run_tests_chunked.py
    • test 1 directory at a time
    • splits known large directories into smaller chunks
  • chrometest - run_chrome_chunked.py
    • make _tests/testing/chrome.bak
    • copies 1 directory at a time from chrome.bak to chrome
  • reftest - reftest_chunked.py
    • parse manifest, run 1 dir at a time
    • for large dir, split into multiple manifests
  • crashtest - reftest_chunked.py
    • parse manifest, run 1 dir at a time
• concat logs into a common log format
  • mochitest - run script to parse each chunk and cat together with summary
  • chrometest - run script to parse each chunk and cat together with summary
  • reftest - run script to remove file:///<rootdir> tags and cat together
  • crashtest - cat log files together
• copy logs off device (leave manual for now)
• report results with new failures highlighted
  • a system to keep track of known failures (json data provider)
  • a system to show results of tests including historical data

I propose writing 2 master scripts and 1 website to deal with this:

• maemkit-packager.py
  • packages all (or one type) the test files from the src and objdir trees
  • creates a .tar file that we can copy
• maemkit-chunked.py - initial draft [here]
  • run a test (or all) as chunked
  • reads a config file to determine
    • what directories to ignore, split
    • how many directories/tests to run
    • which directories/tests to run (if we are running in parallel)
  • create a log file that is compatible for comparison purposes
• maemkit-viewer website
  • provider on known failures
  • provides quick summary of tests
  • provides in depth view of tests
  • provides access to download logs
  • TODO: determine a method for delivering failed test cases and related info to viewer. ctalbert suggests using json and mikeal has a module written for mozmill we might be able to use

NOTE: Ted is working bug 421611 to package mochitest up with make package. Once this is finished, we can look into similar ways for reftests and xpcshell.

Nokia Status Table

	[mochitest]	[chrome]	[reftests]	[crashtests]	[xpcshell]
Run in fennec(tree): linux	Done	Done	Done	Done	Done
Results: fennec(tree) == firefox	[89%]
Investigate linux vs firefox results	[100%]	[100%]	[100%]	[100%]
Run in fennec(install): linux	Done	Done	Done	Done
Results: fennec(install) == fennec(tree)
Investigate maemo vs linux	[100%]	[100%]	[100%]	[100%]
Run on Nokia device (tools basically working)	Done	Done	Done	Done
Run automated on Nokia device	Done	Done	Done	Done
Results: fennec(device) == firefox	[79%]
Documented tools/steps to run on device	[Done]	[Done]	[Done]	[Done]	[Draft]
Initial handoff to build	[Done]
Buildbot running tests

Action Log

• 2009-01-29 - Updated [mochitest] instructions
• 2009-01-28 - Added bug 475778 to figure out how to checkin maemkit
• 2009-01-27 - Completed initial draft of maemkit-chunked
• 2009-01-23 - Updated [Chrome] document with bug 457098 to track fix to mochitest
• 2009-01-23 - coded up [mochitest/chrome] for maemkit-chunked
• 2009-01-15 - Updated maemkit for run_chunked to define config file
• 2009-01-12 - Created maemkit tool section
• 2009-01-07 - Updated [Chrome] document with additional TODO items
• 2009-01-07 - Added chrome investigation at [100%]
• 2009-01-06 - Updated [Chrome] document with TODO items
• 2009-01-05 - Updated [reftest] document with TODO items
• 2009-01-05 - Added crashtest investigation at [100%]
• 2008-12-31 - Added reftest investigations at [100%]
• 2008-12-22 - Added data for mochitest investigations at [100%]
• 2008-12-22 - Added rows for investigating failures/differences
• 2008-12-06 - Update chrome test document with script to run on device
• 2008-12-05 - Created chrome test running page/notes
• 2005-12-05 - Updated reftest/crashtest document to include chunked script for safer running on device
• 2005-12-05 - Updated mochitest document to include chunked script for safer running on device
• 2008-12-04 - Investigating large number of tests which don't run and/or fail in fennec vs. firefox. Appears to be somewhat related to running chunked directories as firefox will have similar failures/skipped tests when running chunked.
• 2008-12-03 - Created firefox vs. fennec log comparison [page]
• 2008-11-13 - Created status/tracking wiki
• 2008-11-12 - Created wiki with reftest instructions
• 2008-11-11 - reftests:Initial run in desktop install, developed script to collect tests and required files into .tar file
• 2008-11-11 - crashtests:Initial run in desktop install, developed script to collect tests and required files into .tar file
• 2008-11-10 - reftests:Initial work done to verify results between fennec and firefox
• 2008-11-10 - [bug 464081] filed to track tests disabled for fennec