CloudServices/Sync/FxSync/Crossweave2

Goals

• JavaScript-ability
• improved error reporting (more finegrained, helpful, etc.)
• be a stable test suite across upcoming refactorings (War On Sync)
• provide reliable perf measurements (track regressions, etc.)

Plan for Crossweave / TPS 2.0

• Rename to Test harness and Profiler for Sync (TPS)

• New docs available at https://developer.mozilla.org/en/TPS

• Python
  • [DONE] refactor using Mozmill 2.0
  • [DONE] remove support for the fx-sync repo and the the XPI version of sync
  • [DONE] improve error handling for timeout conditions and python exceptions
  • [DONE] connect to autolog
  • [DONE] convert config and manifest file format to JSON
  • [DONE] remove result reporting to the MySQL db on brasstacks

• Harness
  • [DONE] clone tps repo from crossweave repo
  • [DONE] refactor tests to JS format
  • [DONE] support JavaScript tests
  • [DONE] refactor verify/verify-not/sync/... to be function calls
  • [DONE] make all test execution async
  • move test files themselves to services/sync/tests/tps

Plan beyond TPS 2.0

• Run on all platforms
• Run against Fennec on mobile hardware (e.g. Tegra boards)
• Perf instrumentation
  • event loop blocking (using MOZ_INSTRUMENT_EVENT_LOOP, see bug 606574; bug 631571 is the related bug for Talos)
  • memory usage (from nsIMemoryReporterManager)
  • GC operations (setting javascript.options.mem.log and grepping the error console for the corresponding log messages would already help; better even use build with --enable-gctimer)
  • JITing (what's trace or method JITed and more importantly, what isn't? no instrumentation yet?!?)
  • object allocations (no instrumentation yet?!?)

rnewman desires

• embed in JS tests? (embedded HTTP server?)
• support fuzz consistency tests
• more operations (restore from bookmarks to test wipe, for example)