CloudServices/Sync/FxSync/Crossweave2: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(21 intermediate revisions by 4 users not shown)
Line 2: Line 2:


* JavaScript-ability
* JavaScript-ability
* improved error reporting (more finegrained, helpful, etc.)
* Improved error reporting (more finegrained, helpful, etc.)
* be a stable test suite across upcoming refactorings (War On Sync)
* Be a stable test suite across upcoming refactorings (War On Sync)
* provide reliable perf measurements (track regressions, etc.)
* Provide reliable perf measurements (track regressions, etc.)
* Better coverage of functionality: reset sync; handling of erroneous server conditions, migrations, etc.; handling of client consistency risks such as unexpected shutdowns…


= Plan =
= Plan for Crossweave / TPS 2.0 =


* Rename to **Test harness and Profiler for Sync** (TPS)
* Rename to '''Test harness and Profiler for Sync''' (TPS)
 
* New docs available at https://developer.mozilla.org/en/TPS


* Python
* Python
** ...
** {{done|refactor}} using Mozmill 2.0
** {{done|remove}} support for the fx-sync repo and the the XPI version of sync
** {{done|improve}} error handling for timeout conditions and python exceptions
** {{done|connect}} to [http://brasstacks.mozilla.com/autolog/ autolog]
** {{done|convert}} config and manifest file format to JSON
** {{done|remove}} result reporting to the MySQL db on brasstacks


* Harness
* Harness
** clone crossweave2 from crossweave repo
** {{done|clone}} [https://hg.mozilla.org/services/tps tps] repo from [https://hg.mozilla.org/automation/crossweave crossweave] repo
** refactor tests to JSON format
** {{done|refactor}} tests to JS format
** support JavaScript tests
** {{done|support}} JavaScript tests
** refactor verify/verify-not/sync/... to be function calls
** {{done|refactor}} verify/verify-not/sync/... to be function calls
** make all test execution async
** {{done|make}} all test execution async
** move test files themselves to services/sync/tests/tps
** move test files themselves to services/sync/tests/tps
= Plan beyond TPS 2.0 =
* Run on all platforms
* Run against Fennec on mobile hardware (e.g. Tegra boards)
* Perf instrumentation
** event loop blocking (using <code>MOZ_INSTRUMENT_EVENT_LOOP</code>, see {{bug|606574}}; {{bug|631571}} is the related bug for Talos)
** memory usage (from <code>nsIMemoryReporterManager</code>)
** GC operations (setting <code>javascript.options.mem.log</code> and grepping the error console for the corresponding log messages would already help; better even use build with <code>--enable-gctimer</code>)
** GC + CC graphs: {{bug|680482}}, https://github.com/amccreight/heapgraph
** JITing (what's trace or method JITed and more importantly, what isn't? no instrumentation yet?!?)
** object allocations (no instrumentation yet?!?)
= rnewman desires =
* embed in JS tests? (embedded HTTP server?)
* support fuzz consistency tests
* more operations (restore from bookmarks to test wipe, for example)
= QA desires =
* port mozmill API on top of TPS, so we can have tests against client side setup wizard work
* create headless TPS, and create javascript/python tests against XPShell
* support TPS against a fully integrated test staging server, with configurable settings (eg. disabling Captcha)
* Support for running on mobile?.  Start with porting and running against desktop fennec

Latest revision as of 20:11, 19 November 2013

Goals

  • JavaScript-ability
  • Improved error reporting (more finegrained, helpful, etc.)
  • Be a stable test suite across upcoming refactorings (War On Sync)
  • Provide reliable perf measurements (track regressions, etc.)
  • Better coverage of functionality: reset sync; handling of erroneous server conditions, migrations, etc.; handling of client consistency risks such as unexpected shutdowns…

Plan for Crossweave / TPS 2.0

  • Rename to Test harness and Profiler for Sync (TPS)
  • Python
    • [DONE] refactor using Mozmill 2.0
    • [DONE] remove support for the fx-sync repo and the the XPI version of sync
    • [DONE] improve error handling for timeout conditions and python exceptions
    • [DONE] connect to autolog
    • [DONE] convert config and manifest file format to JSON
    • [DONE] remove result reporting to the MySQL db on brasstacks
  • Harness
    • [DONE] clone tps repo from crossweave repo
    • [DONE] refactor tests to JS format
    • [DONE] support JavaScript tests
    • [DONE] refactor verify/verify-not/sync/... to be function calls
    • [DONE] make all test execution async
    • move test files themselves to services/sync/tests/tps

Plan beyond TPS 2.0

  • Run on all platforms
  • Run against Fennec on mobile hardware (e.g. Tegra boards)
  • Perf instrumentation
    • event loop blocking (using MOZ_INSTRUMENT_EVENT_LOOP, see bug 606574; bug 631571 is the related bug for Talos)
    • memory usage (from nsIMemoryReporterManager)
    • GC operations (setting javascript.options.mem.log and grepping the error console for the corresponding log messages would already help; better even use build with --enable-gctimer)
    • GC + CC graphs: bug 680482, https://github.com/amccreight/heapgraph
    • JITing (what's trace or method JITed and more importantly, what isn't? no instrumentation yet?!?)
    • object allocations (no instrumentation yet?!?)

rnewman desires

  • embed in JS tests? (embedded HTTP server?)
  • support fuzz consistency tests
  • more operations (restore from bookmarks to test wipe, for example)

QA desires

  • port mozmill API on top of TPS, so we can have tests against client side setup wizard work
  • create headless TPS, and create javascript/python tests against XPShell
  • support TPS against a fully integrated test staging server, with configurable settings (eg. disabling Captcha)
  • Support for running on mobile?. Start with porting and running against desktop fennec
Retrieved from "https://wiki.allizom.org/index.php?title=CloudServices/Sync/FxSync/Crossweave2&oldid=766077"