Mobile/Fennec Automation: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(7 intermediate revisions by the same user not shown)
Line 7: Line 7:
Below is a table outlining what is needed and where we are.  Below the table is a log of actions which is intended for people to see what has been done and when if they want more details about a specific item.
Below is a table outlining what is needed and where we are.  Below the table is a log of actions which is intended for people to see what has been done and when if they want more details about a specific item.


=== Maemkit Toolset ===


NEW: Download the initial drop of [[http://people.mozilla.com/~jmaher/fennec/maemkit/ maemkit-chunked]].
Until we get a lot of changes backported into the build system (makefiles, test running scripts, etc...) we will have a temporary test framework that wraps around the build system and test scripts to run all tests on maemo.  We will call this the maemkit.
Here are a list of things the maemkit needs to do:
* Package the tests
** mochitest: tar -chf $(xr_objdir)/_tests/ $(objdir)/_profile/ tests.tar
** chrometest: same as mochitest
** reftest: run reftest_extract.py
** crashtests: run crashtest_extract.py
* copy to device/unpack (leave manual for now)
* run tests in a chunked method
** mochitest - run_tests_chunked.py
*** test 1 directory at a time
***splits known large directories into smaller chunks
** chrometest - run_chrome_chunked.py
*** make _tests/testing/chrome.bak
*** copies 1 directory at a time from chrome.bak to chrome
** reftest - reftest_chunked.py
*** parse manifest, run 1 dir at a time
*** for large dir, split into multiple manifests
** crashtest - reftest_chunked.py
*** parse manifest, run 1 dir at a time
* concat logs into a common log format
** mochitest - run script to parse each chunk and cat together with summary
** chrometest - run script to parse each chunk and cat together with summary
** reftest - run script to remove file:///<rootdir> tags and cat together
** crashtest - cat log files together
* copy logs off device (leave manual for now)
* report results with new failures highlighted
** a system to keep track of known failures (json data provider)
** a system to show results of tests including historical data
I propose writing 2 master scripts and 1 website to deal with this:
* maemkit-packager.py
** packages all (or one type) the test files from the src and objdir trees
** creates a .tar file that we can copy
* maemkit-chunked.py - initial draft [[http://people.mozilla.com/~jmaher/fennec/maemkit/ here]]
** run a test (or all) as chunked
** reads a config file to determine
*** what directories to ignore, split
*** how many directories/tests to run
*** which directories/tests to run (if we are running in parallel)
** create a log file that is compatible for comparison purposes
* maemkit-viewer website
** provider on known failures
** provides quick summary of tests
** provides in depth view of tests
** provides access to download logs
** TODO: determine a method for delivering failed test cases and related info to viewer.  ctalbert suggests using json and mikeal has a module written for mozmill we might be able to use
NOTE: Ted is working {{bug|421611}} to package mochitest up with make package.  Once this is finished, we can look into similar ways for reftests and xpcshell.


=== Nokia Status Table ===
=== Nokia Status Table ===
Line 102: Line 46:
|Done
|Done
|Done
|Done
|
|Done
|-
|-
|Results: fennec(install) == fennec(tree)
|Results: fennec(install) == fennec(tree)
Line 123: Line 67:
|Done
|Done
|Done
|Done
|
|Done
|-
|-
|Run automated on Nokia device
|Run automated on Nokia device
Line 130: Line 74:
|Done
|Done
|Done
|Done
|
|Done
|-
|-
|Results: fennec(device) == firefox
|Results: fennec(device) == firefox
Line 141: Line 85:
|Documented tools/steps to run on device
|Documented tools/steps to run on device
|[[http://wiki.mozilla.org/Mobile/Fennec_Mochitest Done]]
|[[http://wiki.mozilla.org/Mobile/Fennec_Mochitest Done]]
|[[http://wiki.mozilla.org/Mobile/Fennec_Chrome Draft]]
|[[http://wiki.mozilla.org/Mobile/Fennec_Chrome Done]]
|[[http://wiki.mozilla.org/Mobile/Fennec_Reftests Draft]]
|[[http://wiki.mozilla.org/Mobile/Fennec_Reftests Done]]
|[[http://wiki.mozilla.org/Mobile/Fennec_Reftests Draft]]
|[[http://wiki.mozilla.org/Mobile/Fennec_Reftests Done]]
|
|[[https://wiki.mozilla.org/Mobile/Fennec_Xpcshell Draft]]
|-
|-
|Initial handoff to build
|Initial handoff to build
Line 162: Line 106:


=== Action Log ===
=== Action Log ===
 
* 2009-03-27 - Updated all documentation to run tests on Maemo
* 2009-03-26 - Updated [[http://hg.mozilla.org/qa/maemkit/file/527dc910d556/ maemkit]] code
* 2009-03-23 - Initial drop of maemkit code
* 2009-01-30 - Added [[https://wiki.mozilla.org/Mobile/Fennec_Xpcshell xpcshell]] instructions
* 2009-01-29 - Updated [[https://wiki.mozilla.org/Mobile/Fennec_Mochitest mochitest]] instructions
* 2009-01-29 - Updated [[https://wiki.mozilla.org/Mobile/Fennec_Mochitest mochitest]] instructions
* 2009-01-28 - Added {{bug|475778}} to figure out how to checkin maemkit
* 2009-01-27 - Completed initial draft of maemkit-chunked
* 2009-01-27 - Completed initial draft of maemkit-chunked
* 2009-01-23 - Updated [[https://wiki.mozilla.org/Mobile/Fennec_Chrome#TODO Chrome]] document with {{bug|457098}} to track fix to mochitest
* 2009-01-23 - Updated [[https://wiki.mozilla.org/Mobile/Fennec_Chrome#TODO Chrome]] document with {{bug|457098}} to track fix to mochitest

Latest revision as of 15:20, 18 June 2009

Fennec Automation

In developing Fennec, we need to leverage all the work done for Firefox. As an initial step, we will be porting all the automation that buildbot runs on Firefox and we will run it on Fennec.

There are a lot of new variables as we have non standard devices to test on, tests and tools which don't exist or are invalid in the mobile environment and code which needs to be updated to support a new product and environment.

Below is a table outlining what is needed and where we are. Below the table is a log of actions which is intended for people to see what has been done and when if they want more details about a specific item.


Nokia Status Table

[mochitest] [chrome] [reftests] [crashtests] [xpcshell]
Run in fennec(tree): linux Done Done Done Done Done
Results: fennec(tree) == firefox [89%]
Investigate linux vs firefox results [100%] [100%] [100%] [100%]
Run in fennec(install): linux Done Done Done Done Done
Results: fennec(install) == fennec(tree)
Investigate maemo vs linux [100%] [100%] [100%] [100%]
Run on Nokia device (tools basically working) Done Done Done Done Done
Run automated on Nokia device Done Done Done Done Done
Results: fennec(device) == firefox [79%]
Documented tools/steps to run on device [Done] [Done] [Done] [Done] [Draft]
Initial handoff to build [Done]
Buildbot running tests

Action Log

  • 2009-03-27 - Updated all documentation to run tests on Maemo
  • 2009-03-26 - Updated [maemkit] code
  • 2009-03-23 - Initial drop of maemkit code
  • 2009-01-30 - Added [xpcshell] instructions
  • 2009-01-29 - Updated [mochitest] instructions
  • 2009-01-28 - Added bug 475778 to figure out how to checkin maemkit
  • 2009-01-27 - Completed initial draft of maemkit-chunked
  • 2009-01-23 - Updated [Chrome] document with bug 457098 to track fix to mochitest
  • 2009-01-23 - coded up [mochitest/chrome] for maemkit-chunked
  • 2009-01-15 - Updated maemkit for run_chunked to define config file
  • 2009-01-12 - Created maemkit tool section
  • 2009-01-07 - Updated [Chrome] document with additional TODO items
  • 2009-01-07 - Added chrome investigation at [100%]
  • 2009-01-06 - Updated [Chrome] document with TODO items
  • 2009-01-05 - Updated [reftest] document with TODO items
  • 2009-01-05 - Added crashtest investigation at [100%]
  • 2008-12-31 - Added reftest investigations at [100%]
  • 2008-12-22 - Added data for mochitest investigations at [100%]
  • 2008-12-22 - Added rows for investigating failures/differences
  • 2008-12-06 - Update chrome test document with script to run on device
  • 2008-12-05 - Created chrome test running page/notes
  • 2005-12-05 - Updated reftest/crashtest document to include chunked script for safer running on device
  • 2005-12-05 - Updated mochitest document to include chunked script for safer running on device
  • 2008-12-04 - Investigating large number of tests which don't run and/or fail in fennec vs. firefox. Appears to be somewhat related to running chunked directories as firefox will have similar failures/skipped tests when running chunked.
  • 2008-12-03 - Created firefox vs. fennec log comparison [page]
  • 2008-11-13 - Created status/tracking wiki
  • 2008-11-12 - Created wiki with reftest instructions
  • 2008-11-11 - reftests:Initial run in desktop install, developed script to collect tests and required files into .tar file
  • 2008-11-11 - crashtests:Initial run in desktop install, developed script to collect tests and required files into .tar file
  • 2008-11-10 - reftests:Initial work done to verify results between fennec and firefox
  • 2008-11-10 - [bug 464081] filed to track tests disabled for fennec
Retrieved from "https://wiki.allizom.org/index.php?title=Mobile/Fennec_Automation&oldid=150381"