Auto-tools/Projects/Autolog: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(autolog is dead)
 
(39 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Status ==
Autolog is no longer in service.  It has been superseded by [[Auto-tools/Projects/Treeherder|Treeherder]], a flexible, extensible replacement to tbpl.
== Goal ==
== Goal ==


The Autolog project seeks to implement a TBPL-like system for viewing test results produced by the a-team's various tools, at least those of which aren't hooked up to TBPL.  Such projects potentially include mobile automation, Crossweave, profile manager, etc.
The Autolog project seeks to implement a TBPL-like system for viewing test results produced by the a-team's various tools, at least those of which aren't hooked up to TBPL.  Such projects potentially include mobile automation, Crossweave, profile manager, etc.


== Proposed Implementation ==
== Project Phases ==
 
=== Phase 1 (Q2 2011) ===
 
* front-end
** essentially replicate TBPL, minus the tree-management functions
* back-end
** a python server that queries ElasticSearch and returns data in JSON
** a bugzilla cache to make bugzilla queries fast
** a hg cache to make hg queries fast
** a documented REST API that test tools can use to submit test results
** a python library for python test tools to make result submission easier
** storage for test log files on brasstacks
* integration
** hook up at least one test tool (TPS) to autolog


The system will be comprised of both a front-end web UI, and a backend database.  The database which will be used is ElasticSearch, the same instance of which we're using for OrangeFactor, since our experience with that seems to indicate that it is fast, reliable, and easy to use.
=== Future Phases ===


Communication with the db will be provided by two channels:  a REST API, which we can probably build by extending Orange Factor's woo_server.py, and a python library that automation tools can use to post results to the database.
* front-end
** intelligent display of stack traces, reftest images, and other extended data
** ability to edit, delete orange comments
** display test results by product, instead of just by tree
** better display of TBPL data, including builds and talos runs
** display cumulative stats for tests, possibly by way of OrangeFactor
* back-end
** a cache to reduce load on ElasticSearch and make the ES queries faster
** storage for test log files on a metrics server
** multi-threaded server to handle requests more efficiently


The [http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/ tbpl source] should make a good starting place.  A few files will have to be replaced more-or-less in their entirety, including Data.js, PushlogJSONParser.js, SummaryLoader.js, and TinderboxJSONUser.js.
== Implementation ==


Necessary global changes include the following:
=== Back-end ===
* replacing the notion of 'trees' with 'products' or something similar
** products will be combined together into one log, rather than split between different logs, since we presumably won't be running a lot of simultaneous tests against one product
* test suites are product dependent
** test names and other test metadata will be extracted from the test-run JSON rather than being hardcoded in the client JS


The idea of using CommonJS (specifically, the modules system) was discussed, but given that we will use tbpl as a starting point, it doesn't make sense to redo the basic architecture, which works well. It would also make it difficult to port any new features over to tbpl.
Data is stored in ElasticSearch; the same instance that's used for OrangeFactor.  Data is segregated from WOO data in separate indices.  Data is published to the Autolog ES indices using the [http://hg.mozilla.org/automation/mozautolog/ mozautolog] python library.


asuth of MoMo created an alternative view to tbpl called [http://arbpl.visophyte.org/?tree=Firefox ArbPL] ([https://github.com/asutherland/arbitrarypushlog code]).  Might be interesting for possible UI ideas and as a source of more code that interfaces with tinderbox.
See the data structure section, below.
 
=== Front-end ===
 
The UI is a modified version of TBPL.  Unlike TBPL, The Autolog UI has a server-side component which queries ES and provides data to the Autolog JS code running in the browser.  This is necessary because the ElasticSearch instance being used in inside the MPT network, and can't be queried directly from public browsers.  Using a server-side component provides other benefits, such as allowing us to utilize a Bugzilla cache, and to minimize the amount of data exchanged with the browser by performing some data processing on the server.
 
The Autolog UI displays data published into the Autolog ES indices by default, but can be used to display the buildbot data that is parsed for the War on Orange as well.
 
The Autolog UI is currently running at http://brasstacks.mozilla.com/autolog/.
 
Eventually it might be nice to move the JS code from its current ad-hoc JQuery style to a model-controller-view pattern, using the [[Auto-tools/Projects/WebUXPlatform|Web UX Platform]] .  This would make future maintenance and new features potentially easier to write, although it would also mean any such changes would be impractical to back-port to TBPL.


== Data Structure ==
== Data Structure ==


There are two types of data structures we are concerned with.  One is the structure of data that a test suite will have to provide in order to insert test results into ElasticSearch.  The second is the structure of data inside ElasticSearch itself.
=== Terminology ===
 
'''Testgroup''': a collection of test suites that are run on the same machine as part of the same group.  Each green or orange letter in TBPL is a testgroup.  Each testgroup has one more testsuites.
 
'''Testsuite''':  a group of tests that are run by the same process.
 
In TBPL, mochitest-other is a testgroup, while its constituent tests, like mochitest-a11y and mochitest-ipcplugins, are testsuites.  Most testgroups (like reftest) have only one testsuite.
 
=== Autolog data structure in ElasticSearch ===
 
Autolog data is separated into four different doc_types in ElasticSearch: testgroups, testsuites, testfailures, and perfdata.  All these belong to the 'logs' index.
 
'''Testgroup structure'''
 
{
    buildtype: "opt"
    version: null
    total_test_failures: 0
    buildid: "20110626154522"
    testgroup_id: "1309132817-ca13f590057dddb932141eb837c72ad0820a82d9"
    harness: "buildbot"
    branch: null
    date: "2011-06-27"
    testrun: "4e87265b9c11.1"
    logurl: "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1309128322/mozilla-inbound_leopard_test-jsreftest-build63.txt.gz"
    testgroup: "jsreftest"
    buildurl: "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1309128322/firefox-7.0a1.en-US.mac.dmg"
    builder: "mozilla-inbound_leopard_test-jsreftest"
    tree: "mozilla-inbound"
    productname: null
    machine: "talos-r3-leopard-020"
    platform: "macosx"
    testsuite_count: 1
    starttime: 1309132817
    frameworkfailures: null
    os: "leopard"
    total_perf_records: 0
    pending: false
    revision: "4e87265b9c11"
}
 
'''Testsuite structure'''
 
{
    testsuite: "reftest"
    testgroup: "reftest-ipc"
    starttime: 1309132966
    cmdline: "python reftest/runreftest.py --appname=firefox/firefox-bin --utility-path=bin --extra-profile-file=bin/plugins --symbols-path=symbols --setpref=browser.tabs.remote=true reftest/tests/layout/reftests/reftest-sanity/reftest.list"
    buildid: "20110626160901"
    todo: "12"
    testgroup_id: "1309132966-f71cdc0789f22a84f62aea84cc826b93a266bf1f"
    os: "fedora"
    tree: "mozilla-central"
    elapsedtime: "11"
    failed: "0"
    buildtype: "opt"
    machine: "talos-r3-fed-055"
    platform: "linux"
    passed: "58"
    date: "2011-06-27"
    testsuite_id: "1309132966-f71cdc0789f22a84f62aea84cc826b93a266bf1f-testsuite1"
    testfailure_count: 0
    revision: "6a3e7aebda53"
}


For the former:
'''Testfailure structure'''


  {
  {
  // testgroup definition
    buildtype: "opt"
  'harness': 'tinderbox',
    errors: [
  'testgroup': 'mochitest-other',
        {
  'machine': 'talos-r3-fed64-044',
            status: "TEST-UNEXPECTED-FAIL"
  'testsuite_count': 1,          // supplied by python lib
            text: "test failed (with xpcshell return code: 0), see following log:"
  'starttime': 1297879654,
        },
  'date': '2011-02-16',          // supplied by python lib
        {
  'logurl': '...',                // optional
            status: "TEST-UNEXPECTED-FAIL"
  'os': 'fedora12',
            text: "2147746065"
  'platform': 'linux',
        }
  // 'testrun' is an implementation-specific identifier which
    ]
  // is unique among a set of related testgroups; it is used to
    testfailure_id: "1309132818-6476508d66a9686a94e1d1c1b8671f5f73125e76-testfailure1.1"
  // differentiate multiple sets of testgroups which may be run
    buildid: "20110626154522"
  // against the same changeset
    os: "leopard"
  'testrun': '...',
    testgroup_id: "1309132818-6476508d66a9686a94e1d1c1b8671f5f73125e76"
 
    tree: "mozilla-inbound"
  // Base product definition: the primary product under test.  For
    machine: "talos-r3-leopard-013"
  // Crossweave, this is the fx-sync code; for Android it is
    platform: "macosx"
  // the mobile-browser code.  In a TBPL-like display, this
    starttime: 1309132818
  // product's rev would be displayed in the "commit" column.
    date: "2011-06-27"
  'productname': 'sync',
    test: "xpcshell/tests/services/sync/tests/unit/test_syncengine_sync.js"
  'tree': 'fx-sync',
    testgroup: "xpcshell"
  'branch': '1.7',                // optional?
    revision: "4e87265b9c11"
  'revision': '553f7b1974a3',
    testsuite_id: "1309132818-6476508d66a9686a94e1d1c1b8671f5f73125e76-testsuite1"
  'buildtype': 'xpi',
    logurl: null
  'buildid': '20110210030206',    // optional
  'version': '1.7.pre',          // optional
  'buildurl': '...',              // optional
 
  // Secondary product definitions:  additional products involved
  // in the test.  For Crossweave or Android, this might be
  // 'mozilla-central', etc. There can be as many secondary
  // products as needed.
  'tree2': 'mozilla-central',
  'branch2': 'default',          // optional?
  'revision2': '553f7b1974a3',    // optional for secondary products
  'buildtype2': 'opt',
  'buildid2': '20110210030206',  // optional
  'version2': '4.0b13pre',        // optional
  'buildurl2': '...',            // optional
 
  // Testsuite definition.  This is an array (to support cases like
  // mochitest-other); only one member is shown in the example below.
  'testsuites': [
    // for cases other than mochitest-other, this is probably the
    // same as 'testgroup' above
    'suitename': 'mochitest-ally1',
    'cmdline': '...',            // optional
    'testfailure_count': 1,      // provided by python lib
    'elapsedtime': 665,          // in seconds
    'passed': 85152,
    'failed': 1,
    'todo': 124,
   
    // These are failures that occur during specific test cases.
    'testfailures': [
      // 'test' is null in cases where the error cannot be assigned to a specific test case,
      // e.g., crashes that occur after all tests have finished
      'test': 'xpcshell/tests/toolkit/components/places/tests/autocomplete/test_download_embed_bookmarks.js',
      // per test logs, optional
      'logurl': '...',
      // Like testsuite errors, each member of 'failures' can contain
      // additional metadata depending on failure type.
      'failures': [
        'status': 'TEST-UNEXPECTED-FAIL',
        'text': 'Acceleration enabled on Windows XP or newer - didn't expect 0, but got it'
      ]
    ],
  ]
  }
  }


For the structure in ElasticSearch, the data will be separated into three document types (by the python library if that's used; the test suite will have to do this if it's posting to ES directly via HTTP), similar to the way that tinderbox logs are spread across three document types at present:
'''Perfdata structure'''


* a testgroup document (corresponding to tinderbox build documents, example [http://elasticsearch1.metrics.sjc1.mozilla.com:9200/logs/builds/_search?q=date:2011-02-18 here])
{
* one or more testsuite documents (corresponding to tinderbox testrun documents, example [http://elasticsearch1.metrics.sjc1.mozilla.com:9200/logs/testruns/_search?q=date:2011-02-10 here])
    buildtype: "debug"
* one or more testfailure documents (corresponding to tinderbox testfailure documents, example [http://elasticsearch1.metrics.sjc1.mozilla.com:9200/logs/testfailures/_search?q=date:2011-02-10 here])
    testgroup: "xperfstartup"
    platform: "win32"
    buildid: "20110216155739"
    os: "win7"
    testgroup_id: "2xj4Fex1Qh-_H1g3kfpnRw"
    tree: "mozilla-central"
    machine: "test"
    perfdata: [
        {
            read_bytes: 1085440
            type: "diskIO"
            name: "\Device\HarddiskVolume1\test0511\firefox\xul.dll"
            reads: 44
        },
        {
            count: 314
            type: "pagefaults"
            name: "firefox.exe"
        },
        {
            count: 314159
            type: "pagefaults"
            name: "system"
        }
    ]
    starttime: 1297900396
    date: "2011-02-16"
    test: "xperfstartup"
    testsuite: "xperfstartup"
    testsuite_id: "4xaGI6ejSgSTWL0x3eENpw"
    revision: "0f777e59d48c"
}


'''Q''': Why do we separate the data into three document types, why not just use one big document?<br/>
'''Q''': Why do we separate the data into three document types, why not just use one big document?<br/>
Line 119: Line 202:
'''Q''': Is this really the best way to include data about multiple products, or code from multiple repos?<br/>
'''Q''': Is this really the best way to include data about multiple products, or code from multiple repos?<br/>
'''A''': I'm not sure.  I suggested this structure because it's easy to use when searching ES.  Other structures are possible.  For instance, we could create a 'product' document type, and store all the products there, and then just include indexes to this document in the 'testgroup' document.  The downside to this is that getting certain data out of ES would require multiple queries.
'''A''': I'm not sure.  I suggested this structure because it's easy to use when searching ES.  Other structures are possible.  For instance, we could create a 'product' document type, and store all the products there, and then just include indexes to this document in the 'testgroup' document.  The downside to this is that getting certain data out of ES would require multiple queries.
=== Open Issues ===
==== Identifying 'testruns' ====
The above structure would work fine for displaying a TBPL-like result view.  It might be problematic if we intend to feed into OrangeFactor, however.
The problem is in identifying unique test runs.  For OrangeFactor, we rely on the fact the buildbot uses the same buildid with all related 'testgroups'.  If buildbot reruns the same testgroups (because it's bored), it generates a new buildid, even though the revision is still the same.  Thus we can identify unique test runs.
For non-buildbot cases (and I'm specifically thinking of Crossweave), we don't have an analogous buildid.  In the Crossweave case, each 'testgroup' fired off against a given revision is independent and doesn't share any metadata (like a buildid) with other testgroups run at the same time.
If we want to maintain consistency with OrangeFactor, we may have to require a 'buildid' value, which would be the same across all testgroups which are initiated by the same event.  This would require some refactoring of Crossweave and possibly other tools.
Using 'buildid' isn't a perfect solution, though, as some buildbot jobs (like once-a-day win64 runs) use a distinct buildid that doesn't match other buildbot jobs run on the same changeset.  We're currently excluding these in OrangeFactor so it isn't skewing data, but it illustrates the drawbacks of relying on buildid.
The other option is to change the way OrangeFactor identifies testruns. Instead of relying on buildid, we could implement some algorithm like this:
# identify all the changesets that have testgroups; the number of changesets is our preliminary testrun_count
# for each changeset, identify the list of testgroups (L) and the unique set of testgroups (S), based on 'testgroup' and 'platform'
# if len(S) > len(L), create a list of 'extra testgroups' (E) by removing members of (S) from (L)
# sort (E) and look for the maximum number of duplicate entries (e.g., if (E) contains three 'mochitest-other'/'win32' and two 'mochitest-1/5'/'linux64', return 3), and add this number of the preliminary testrun_count
There's always going to be some guesswork involved, since the idea of a 'testrun' we're using in OrangeFactor is entirely an intellectual concept and not something supported intrinsically by our test frameworks.
'''update''': Per the WOO meeting on 2011-03-02, we decided to add a 'testrun' field to the data structure above, which the implementation would be responsible for filling with a value that is unique for specific collections of testgroups.
==== Log Storage ====
Some tools will need to store logs somewhere that can be served by autolog on request; we can't store these in ES, so where should they go?  Options include stage, brasstacks, or some alternate solution provided by metrics.  In order to engage other teams, we'll likely need guesstimates about total storage needed per month and some idea about the retention policy.


=== TBPL Data Structure ===
=== TBPL Data Structure ===
Line 230: Line 284:
All of this gets fed into UserInterface.js in the handleUpdatedPush function().
All of this gets fed into UserInterface.js in the handleUpdatedPush function().


== UI ==
== Publishing data into Autolog ==


There are three main aspects of TBPL that we want to present in Autolog:
Tools wishing to publish data into Autolog should use the mozautolog python library (http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/).  See documentation at http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/raw-file/tip/README.html.


# a list of on-going test runs
A simple script that uploads some (static, for the sake of clarity) performance data to Autolog might look like:
# a colour-coded list of (abbreviated) names of on-going test suites for each test run
# popups with detailed test data


The first two are dynamically updated, with newly started runs being inserted at the top, and test suites changing colours as they change states.
  from mozautolog import RESTfulAutologTestGroup
 
 
Additional, useful information:
  def main():
 
    testgroup = RESTfulAutologTestGroup(
# list of currently failing tests
      testgroup = 'xperfstartup',
# filters
      os = 'win7',
# dropdowns for metadata (abbreviations, tree info, etc.)
      platform = 'win32',
 
      machine = 'test',
Since TBPL's UI is pretty clean and crisp, we should try to reuse TBPL's HTML and CSS.  JavaScript might be more problematic, depending on how closely tied it is to the underlying layers.
      starttime = 1297900397,
 
      builder = 'mozilla-central_win7-debug_test-xperfstartup',
== Client/Server ==
      server = '127.0.0.1:9200',
 
      restserver = 'http://127.0.0.1:8051/'
TBPL puts all the work into the client. The client-side JavaScript is responsible for querying tinderbox directly.
    )
 
    testgroup.set_primary_product(
Autolog will have a relatively thin server side with a caching layer to limit the number of queries going to the ES database. Since the UI will only display a subset of the total data for each test run, the smaller set can be cached and refreshed periodically.  Requested for test-run details will still go directly to the ES database.
      tree = 'mozilla-central',
      buildtype = 'debug',
      buildid = '20110216155739',
      revision = '0f777e59d48c',
    )
    testgroup.add_perf_data(
      test = 'xperfstartup',
      type = 'diskIO',
      name = '\\Device\\HarddiskVolume1\\test0511\\firefox\\xul.dll',
      reads = 44,
      read_bytes = 1085440
    )
    testgroup.add_perf_data(
      test = 'xperfstartup',
      type = 'pagefaults',
      name = 'firefox.exe',
      count = 314
    )
    testgroup.add_perf_data(
      test = 'xperfstartup',
      type = 'pagefaults',
      name = 'system',
      count = 314159
    )
    testgroup.submit()
 
  if __name__ == '__main__':
    main()


== Setting up a Development Environment ==
== Setting up a Development Environment ==
Line 258: Line 337:
Pre-requisites:
Pre-requisites:


* pyes (http://pypi.python.org/packages/source/p/pyes/pyes-0.14.1.tar.gz, or easy_install pyes)
* pyes 0.15 (easy_install http://pypi.python.org/packages/source/p/pyes/pyes-0.15.0.tar.gz)
** easy_install requires the python-dev package on linux
* mozautoeslib (http://hg.mozilla.org/automation/mozautoeslib/)
* mozautoeslib (http://hg.mozilla.org/automation/mozautoeslib/)
* mozautolog (http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/)
* bzcache (http://hg.mozilla.org/users/jgriffin_mozilla.com/bzcache)
* dateutil (pip install python-dateutil)
* yaml (pip install pyYAML)
* webob (pip install webob)
* autolog (http://hg.mozilla.org/automation/autolog/)
* autolog (http://hg.mozilla.org/automation/autolog/)
** requires webob and paste, available in the python-webob and python-paste packages in Ubuntu


Steps:
Steps:


# Setup a local instance of ElasticSearch for development purposes, and populate it with test data, see [http://hg.mozilla.org/automation/autolog/file/tip/README-testdata.txt README-testdata.txt].  By default, this will operate on http://localhost:9200/
# Setup a local instance of ElasticSearch for development purposes. (You can optionally populate it with test data, see [http://hg.mozilla.org/automation/autolog/file/tip/README-testdata.txt README-testdata.txt], but this is broken as of 2012-11-28).  By default, this will operate on http://localhost:9200/
# Start the autolog server in the autolog repo, using <code>python autolog_server.py .</code>
# Edit autolog_server.conf, changing both es_server and bz_cache_server in [autolog] to localhost:9200
# Host the autolog repo using a webserver; I use Apache but presumably nginx or anything else would work equally well.
# Edit js/Config.js, changing autologServer attribute of the Config object to "http://localhost:8051/". You may also need to add extra IDs to the OSNames.
# Navigate to index.html in the autolog repo; depending on how you've configured your webserver this might look something like http://localhost/autolog/
# Start the autolog server in the autolog repo, using <code>python autolog_server.py .</code> (yes, include the dot at the end)
# Host the autolog repo using a webserver; I use Apache but presumably nginx or anything else would work equally well. Alternatively, you can load it locally with a file URL.
# Navigate to index.html in the autolog repo; depending on how you've configured your webserver this might look something like http://localhost/autolog/ or file:///path/to/autolog/index.html.


Notes:
Notes:
Line 276: Line 364:
   python testdata.py
   python testdata.py


== Tasks ==
* You can view the raw data returned by the autolog server using this url:  http://localhost:8080/testgroups
 
{| class="wikitable" style="width: 100%" class="fullwidth-table sortable"
|+
| Task || Owner || Notes
|-
| Investigate TBPL code and determine how much to re-use
| mcote
| We will use it as a starting point, stripping out what's unnecessary and modifying existing code as necessary to deal with autolog's (simpler) data source.
|-
| Client code (HTML/CSS/JS)
| mcote, jmaher?
|
|-
| Propose common data structure for test results in db
| Jgriffin
|
|-
| Clean up woo_server.py and modularize in order to make future additions easier
| ?
|
|-
| Create python library for test tools to use to post results to db
| jgriffin
|
|-
| Investigate implementing threading in woo_server
|
|
|}

Latest revision as of 19:26, 9 February 2015

Status

Autolog is no longer in service. It has been superseded by Treeherder, a flexible, extensible replacement to tbpl.

Goal

The Autolog project seeks to implement a TBPL-like system for viewing test results produced by the a-team's various tools, at least those of which aren't hooked up to TBPL. Such projects potentially include mobile automation, Crossweave, profile manager, etc.

Project Phases

Phase 1 (Q2 2011)

  • front-end
    • essentially replicate TBPL, minus the tree-management functions
  • back-end
    • a python server that queries ElasticSearch and returns data in JSON
    • a bugzilla cache to make bugzilla queries fast
    • a hg cache to make hg queries fast
    • a documented REST API that test tools can use to submit test results
    • a python library for python test tools to make result submission easier
    • storage for test log files on brasstacks
  • integration
    • hook up at least one test tool (TPS) to autolog

Future Phases

  • front-end
    • intelligent display of stack traces, reftest images, and other extended data
    • ability to edit, delete orange comments
    • display test results by product, instead of just by tree
    • better display of TBPL data, including builds and talos runs
    • display cumulative stats for tests, possibly by way of OrangeFactor
  • back-end
    • a cache to reduce load on ElasticSearch and make the ES queries faster
    • storage for test log files on a metrics server
    • multi-threaded server to handle requests more efficiently

Implementation

Back-end

Data is stored in ElasticSearch; the same instance that's used for OrangeFactor. Data is segregated from WOO data in separate indices. Data is published to the Autolog ES indices using the mozautolog python library.

See the data structure section, below.

Front-end

The UI is a modified version of TBPL. Unlike TBPL, The Autolog UI has a server-side component which queries ES and provides data to the Autolog JS code running in the browser. This is necessary because the ElasticSearch instance being used in inside the MPT network, and can't be queried directly from public browsers. Using a server-side component provides other benefits, such as allowing us to utilize a Bugzilla cache, and to minimize the amount of data exchanged with the browser by performing some data processing on the server.

The Autolog UI displays data published into the Autolog ES indices by default, but can be used to display the buildbot data that is parsed for the War on Orange as well.

The Autolog UI is currently running at http://brasstacks.mozilla.com/autolog/.

Eventually it might be nice to move the JS code from its current ad-hoc JQuery style to a model-controller-view pattern, using the Web UX Platform . This would make future maintenance and new features potentially easier to write, although it would also mean any such changes would be impractical to back-port to TBPL.

Data Structure

Terminology

Testgroup: a collection of test suites that are run on the same machine as part of the same group. Each green or orange letter in TBPL is a testgroup. Each testgroup has one more testsuites.

Testsuite: a group of tests that are run by the same process.

In TBPL, mochitest-other is a testgroup, while its constituent tests, like mochitest-a11y and mochitest-ipcplugins, are testsuites. Most testgroups (like reftest) have only one testsuite.

Autolog data structure in ElasticSearch

Autolog data is separated into four different doc_types in ElasticSearch: testgroups, testsuites, testfailures, and perfdata. All these belong to the 'logs' index.

Testgroup structure 

{
   buildtype: "opt"
   version: null
   total_test_failures: 0
   buildid: "20110626154522"
   testgroup_id: "1309132817-ca13f590057dddb932141eb837c72ad0820a82d9"
   harness: "buildbot"
   branch: null
   date: "2011-06-27"
   testrun: "4e87265b9c11.1"
   logurl: "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1309128322/mozilla-inbound_leopard_test-jsreftest-build63.txt.gz"
   testgroup: "jsreftest"
   buildurl: "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1309128322/firefox-7.0a1.en-US.mac.dmg"
   builder: "mozilla-inbound_leopard_test-jsreftest"
   tree: "mozilla-inbound"
   productname: null
   machine: "talos-r3-leopard-020"
   platform: "macosx"
   testsuite_count: 1
   starttime: 1309132817
   frameworkfailures: null
   os: "leopard"
   total_perf_records: 0
   pending: false
   revision: "4e87265b9c11"
}

Testsuite structure 

{
   testsuite: "reftest"
   testgroup: "reftest-ipc"
   starttime: 1309132966
   cmdline: "python reftest/runreftest.py --appname=firefox/firefox-bin --utility-path=bin --extra-profile-file=bin/plugins --symbols-path=symbols --setpref=browser.tabs.remote=true reftest/tests/layout/reftests/reftest-sanity/reftest.list"
   buildid: "20110626160901"
   todo: "12"
   testgroup_id: "1309132966-f71cdc0789f22a84f62aea84cc826b93a266bf1f"
   os: "fedora"
   tree: "mozilla-central"
   elapsedtime: "11"
   failed: "0"
   buildtype: "opt"
   machine: "talos-r3-fed-055"
   platform: "linux"
   passed: "58"
   date: "2011-06-27"
   testsuite_id: "1309132966-f71cdc0789f22a84f62aea84cc826b93a266bf1f-testsuite1"
   testfailure_count: 0
   revision: "6a3e7aebda53"
}

Testfailure structure 

{
   buildtype: "opt"
   errors: [
       {
           status: "TEST-UNEXPECTED-FAIL"
           text: "test failed (with xpcshell return code: 0), see following log:"
       },
       {
           status: "TEST-UNEXPECTED-FAIL"
           text: "2147746065"
       }
   ]
   testfailure_id: "1309132818-6476508d66a9686a94e1d1c1b8671f5f73125e76-testfailure1.1"
   buildid: "20110626154522"
   os: "leopard"
   testgroup_id: "1309132818-6476508d66a9686a94e1d1c1b8671f5f73125e76"
   tree: "mozilla-inbound"
   machine: "talos-r3-leopard-013"
   platform: "macosx"
   starttime: 1309132818
   date: "2011-06-27"
   test: "xpcshell/tests/services/sync/tests/unit/test_syncengine_sync.js"
   testgroup: "xpcshell"
   revision: "4e87265b9c11"
   testsuite_id: "1309132818-6476508d66a9686a94e1d1c1b8671f5f73125e76-testsuite1"
   logurl: null
}

Perfdata structure 

{
   buildtype: "debug"
   testgroup: "xperfstartup"
   platform: "win32"
   buildid: "20110216155739"
   os: "win7"
   testgroup_id: "2xj4Fex1Qh-_H1g3kfpnRw"
   tree: "mozilla-central"
   machine: "test"
   perfdata: [
       {
           read_bytes: 1085440
           type: "diskIO"
           name: "\Device\HarddiskVolume1\test0511\firefox\xul.dll"
           reads: 44
       },
       {
           count: 314
           type: "pagefaults"
           name: "firefox.exe"
       },
       {
           count: 314159
           type: "pagefaults"
           name: "system"
       }
   ]
   starttime: 1297900396
   date: "2011-02-16"
   test: "xperfstartup"
   testsuite: "xperfstartup"
   testsuite_id: "4xaGI6ejSgSTWL0x3eENpw"
   revision: "0f777e59d48c"
}

Q: Why do we separate the data into three document types, why not just use one big document?
A: Because searches in ElasticSearch are must faster and easier with basic data types; searching inside complex nested JSON is slower and the syntax is much more complex.

Q: Can't the python library automatically provide 'os' and 'platform'?
A: It would be nice, wouldn't it? Unfortunately, there are lots of things which can confuse the issue; e.g., if you're using mozilla-build on Windows, it will see your 64-bit version of Windows as win32, regardless of what you're testing. Similarly, we sometimes test 32-bit Mac stuff on macosx64. It seems safest to have the test tools provide this data instead of trying to guess.

Q: Why do we have both testgroup and testsuite?
A: It's entirely to support mochitest-other.  :( In most cases, each testgroup will have 1 testsuite.

Q: Where are the test runs in this structure?
A: We've been using the term 'testrun' to mean different things in different places. In this structure, I imagine 'testrun' to mean the same thing as it does in OrangeFactor: that is, a collection of testgroups that are run against the same primary changeset.

Q: Is this really the best way to include data about multiple products, or code from multiple repos?
A: I'm not sure. I suggested this structure because it's easy to use when searching ES. Other structures are possible. For instance, we could create a 'product' document type, and store all the products there, and then just include indexes to this document in the 'testgroup' document. The downside to this is that getting certain data out of ES would require multiple queries.

TBPL Data Structure

The basic unit of data in TBPL is the push. A push according to TBPL looks like this: 

"b853c6efa929": {
 "id": 19218,
 "pusher": "dougt@mozilla.com",
 "date": "2011-03-17T20:50:37.000Z",
 "toprev": "b853c6efa929",
 "defaultTip": "b853c6efa929",
 "patches": [
  {
   "rev": "b853c6efa929",
   "author": "Doug Turner",
   "desc": "Bug 642291 - crash [@ nsBufferedInputStream::Write] demos.mozilla.org motovational poster. ipc
           serialization does not work here, removing it. r=bent a=blocking-fennec",
   "tags": {
    "length": 0,
    "prevObject": {
     "length": 0
    }
   }
  }
 ]
},

Additionally, each push can have a 'results' key, which contains all the results associated with that push. If the 'results' key exists, it looks like this: 

'results': {
  'linux': {
    'opt': {
      'Reftest': [ an array of machineResults ],
      'Mochitest': [ an array of machineResults ],
      etc,
    },
    'debug': {}
  },
  'linux64': {}, etc
}

Each 'machineResult' looks like this: 

"1300280775.1300281487.29409.gz": {
 "tree": "Firefox",
 "machine": {
  "name": "Rev3 WINNT 5.1 mozilla-central opt test mochitests-2/5",
  "os": "windowsxp",
  "type": "Mochitest",
  "debug": false,
  "latestFinishedRun": (a reference to the last finished run for this machine),
  "runs": 0,
  "runtime": 0,
  "averageCycleTime": 0
 },
 "slave": "talos-r3-xp-039",
 "runID": "1300280775.1300281487.29409.gz",
 "state": "success",
 "startTime": "2011-03-16T13:06:15.000Z",
 "endTime": "2011-03-16T13:19:01.000Z",
 "briefLogURL": "http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1300280775.1300281487.29409.gz",
 "fullLogURL": "http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1300280775.1300281487.29409.gz&fulltext=1",
 "summaryURL": "php/getSummary.php?tree=Firefox&id=1300280775.1300281487.29409.gz",
 "revs": {
  "mozilla-central": "ee18eff42c2e"
 },
 "notes": [],
 "errorParser": "unittest",
 "_scrape": [
  " s: talos-r3-xp-039",
  "<a href=http://hg.mozilla.org/mozilla-central/rev/ee18eff42c2e title=\"Built from revision  e18eff42c2e\">rev:ee18eff42c2e</a>",
  " mochitest-plain-2
11855/0/292"
 ]
 'push': (reference to the push this belongs to),
 'getTestResults': a function,
 'getScrapeResults': a function,
 'getUnitTestResults': a function,
 'getTalosResults': a function,
},

All of this gets fed into UserInterface.js in the handleUpdatedPush function().

Publishing data into Autolog

Tools wishing to publish data into Autolog should use the mozautolog python library (http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/). See documentation at http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/raw-file/tip/README.html.

A simple script that uploads some (static, for the sake of clarity) performance data to Autolog might look like: 

 from mozautolog import RESTfulAutologTestGroup
 
 def main():
   testgroup = RESTfulAutologTestGroup(
     testgroup = 'xperfstartup',
     os = 'win7',
     platform = 'win32',
     machine = 'test',
     starttime = 1297900397,
     builder = 'mozilla-central_win7-debug_test-xperfstartup',
     server = '127.0.0.1:9200',
     restserver = 'http://127.0.0.1:8051/'
   )
   testgroup.set_primary_product(
     tree = 'mozilla-central',
     buildtype = 'debug',
     buildid = '20110216155739',
     revision = '0f777e59d48c',
   )
   testgroup.add_perf_data(
     test = 'xperfstartup',
     type = 'diskIO',
     name = '\\Device\\HarddiskVolume1\\test0511\\firefox\\xul.dll',
     reads = 44,
     read_bytes = 1085440
   )
   testgroup.add_perf_data(
     test = 'xperfstartup',
     type = 'pagefaults',
     name = 'firefox.exe',
     count = 314
   )
   testgroup.add_perf_data(
     test = 'xperfstartup',
     type = 'pagefaults',
     name = 'system',
     count = 314159
   )
   testgroup.submit()
 
 if __name__ == '__main__':
   main()

Setting up a Development Environment

Pre-requisites:

Steps:

  1. Setup a local instance of ElasticSearch for development purposes. (You can optionally populate it with test data, see README-testdata.txt, but this is broken as of 2012-11-28). By default, this will operate on http://localhost:9200/
  2. Edit autolog_server.conf, changing both es_server and bz_cache_server in [autolog] to localhost:9200
  3. Edit js/Config.js, changing autologServer attribute of the Config object to "http://localhost:8051/". You may also need to add extra IDs to the OSNames.
  4. Start the autolog server in the autolog repo, using python autolog_server.py . (yes, include the dot at the end)
  5. Host the autolog repo using a webserver; I use Apache but presumably nginx or anything else would work equally well. Alternatively, you can load it locally with a file URL.
  6. Navigate to index.html in the autolog repo; depending on how you've configured your webserver this might look something like http://localhost/autolog/ or file:///path/to/autolog/index.html.

Notes:

  • The test data is inserted into the local ES using relative dates, i.e., "5 minutes ago". If you are testing code the day after you added the test data, you might want to reset the test data so that it appears with more recent dates. To do so, use these commands (from the autolog repo):
 python testdata.py --wipe
 python testdata.py
Retrieved from "https://wiki.allizom.org/index.php?title=Auto-tools/Projects/Autolog&oldid=1054809"