MozillaWiki

Auto-tools/Projects/OrangeFactor: Difference between revisions

Page Discussion

Auto-tools/Projects/OrangeFactor (view source)

Revision as of 09:07, 7 June 2018

2,420 bytes removed ,  7 June 2018
m
Make decommission note larger
m (Edmorley moved page Auto-tools/Projects/WarOnOrange to Auto-tools/Projects/OrangeFactor: Renaming to the actual name of the webapp)
m (Make decommission note larger)
 
(10 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== Status ==
= OrangeFactor has been decommissioned =
'''It has been succeeded by [[EngineeringProductivity/Projects/IntermittentFailuresView|Intermittent Failures View]].'''


The [http://brasstacks.mozilla.com/orangefactor/ War on Orange web app] provides a variety of ways to view data on buildbot orange failures.  File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Orange%20Factor Testing/Orange Factor].
== About ==


We were taking [[Auto-tools/Projects/WarOnOrange/Meetings|minutes of meetings]] early on in the project, but since then we are tracking features and bugs in [https://bugzilla.mozilla.org/buglist.cgi?list_id=3163513;resolution=---;query_format=advanced;component=Orange%20Factor;product=Testing Bugzilla].
The [https://brasstacks.mozilla.com/orangefactor/ OrangeFactor] web app is a tool for tracking and analysing intermittent test failures encountered that occur during Firefox/gecko continuous integration automation.


For an overview, see [http://people.mozilla.com/~mcote/war-on-orange/war-on-orange-paper-testistanbul.pdf this paper] and the accompanying [http://people.mozilla.com/~mcote/war-on-orange/war-on-orange-testistanbul-slides-presentation.pdf slides], prepared for the TestIstanbul 2012 conference.
NB: OrangeFactor is considered near end of life. A replacement will likely use Treeherder's API as a backend instead.


== Goals ==
== Contributing ==


'''Primary Goal'''
For help with OrangeFactor, please contact #ateam or :emorley, :gbrown, :jmaher.


To develop a web dashboard that is useful for identifying and tracking the state of intermittent oranges in our tinderbox unit tests. This should help developers identify which oranges are most 'interesting', and should give people a notion of the overall state of oranges over time.
File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=OrangeFactor Tree Management::OrangeFactor]. See open bugs [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&component=OrangeFactor&product=Tree+Management here]


'''Secondary Goals'''
The OrangeFactor web app can be run locally. See the instructions at:
 
https://hg.mozilla.org/automation/orangefactor/file/default/README.txt
Since the implementation of the dashboard will require tinderbox failures be put into a database, we could potentially use this database in the tinderbox+pushlog UI, which would allow it to query data from a (fast) database, rather than parsing buildbot logs as it sometimes currently does.
 
== History ==
 
These projects are deprecated and replaced by the new War on Orange/Orange Factor application.
 
'''Topfails'''
 
[http://brasstacks.mozilla.com/topfails/ Topfails] was the first database-driven orange tracker developed in our team.  It shows failures in terms of overall occurrences.  It suffers from a buggy log parser, and a UI with relatively few views.
 
source: http://hg.mozilla.org/automation/topfails/
 
'''Old Orange Factor'''
 
[http://jmaher.couchone.com/orange_factor/_design/woo/orange.html Orange Factor] is a newer dashboard by jmaher.  It calculates the average number of oranges per push (the 'orange factor'), and tracks that number over time.  We're currently using it as a base to explore the usefulness of other statistics.
 
source: http://github.com/jmaher/Orange-Factor
 
[[Auto-tools/Projects/WarOnOrange/InstallationInstructions|Installation instructions]]
 
[https://bugzilla.mozilla.org/buglist.cgi?resolution=---&component=Orange%20Factor&product=Testing bugs]


== Architecture ==
== Architecture ==


The system has several moving parts:
* [https://brasstacks.mozilla.com/ brasstacks.mozilla.com]
** SSL termination occurs on the box.
** Only listens on port 443, since HTTP->HTTPS redirection performed by Zeus.


* modifications to TBPL that write orange comments to a database
* [https://brasstacks.mozilla.com/orangefactor/ OrangeFactor UI]:
** Static UI that interacts with the OrangeFactor API.
** Served by [https://openresty.org/ OpenResty].
** [https://groups.google.com/forum/#!topic/mozilla.dev.tree-management/cIc63kVSN5M Requires LDAP credentials to access].
** [https://hg.mozilla.org/automation/orangefactor/file/default/html Source].


* a [http://pulse.mozilla.org/ Mozilla Pulse] consumer that listens for buildbot messages that are generated when unit tests are finished
* [https://brasstacks.mozilla.com/orangefactor/api/ OrangeFactor REST API]:
** Python FastCGI app reverse proxied by OpenResty.
** [https://groups.google.com/forum/#!topic/mozilla.dev.tree-management/cIc63kVSN5M Requires LDAP credentials to access].
** Read-only API apart from job classification submissions from [https://github.com/mozilla/treeherder Treeherder], which use [https://github.com/kumar303/mohawk Hawk authentication].
** API responses are a combination of results from OrangeFactor's ES instance and the public hg.mozilla.org pushlog.
** ES queries are made using [https://github.com/aparo/pyes pyes], plus a helper library we've written on top of it, [https://hg.mozilla.org/automation/mozautoeslib/ mozautoeslib].
** [https://hg.mozilla.org/automation/orangefactor/file/default/server Source].


* a unittest [http://hg.mozilla.org/automation/logparser/ logparser], that parses buildbot logs, and feeds the resulting data into ElasticSearch
* OrangeFactor Elasticsearch instance:
** Index `bugs`: The intermittent test failure records submitted by Treeherder via OrangeFactor's REST API.
** Index `bzcache`: A cache of public `keyword:intermittent-failure` bugs populated via a brasstacks cron job.


* an instance of [http://www.elasticsearch.org/ ElasticSearch], which is hosted by the Metrics team, that stores the parsed log data and the TBPL bug data
* OrangeFactor bzcache refresh task:
** brasstacks cron job run every four hours, that populates the `bzcache` index on the OrangeFactor Elasticsearch instance.
** Fetches bug data using unauthenticated requests to Bugzilla's REST API.
** [https://github.com/jonallengriffin/bzcache/blob/master/bzcache/bz_cache_refresh.py Source].


* a [http://brasstacks.mozilla.com/orangefactor/ web dashboard] that pulls data from the database and displays various interesting statistics about it
* OrangeFactor mailer task:
** brasstacks cron job <del>run every week</del> now disabled ({{bug|1426416}}), that emails a summary of top failures to [https://groups.google.com/forum/#!forum/mozilla.dev.tree-alerts mozilla.dev.tree-alerts].
** Fetches data from the OrangeFactor REST API.
** [https://hg.mozilla.org/automation/orangefactor/file/default/woo_mailer.py Source].


'''Note:''' The system no longer depends upon flume for writing data into ES.  Instead, we use [https://github.com/aparo/pyes pyes], plus a helper library we've written on top of it, [http://hg.mozilla.org/automation/mozautoeslib/ mozautoeslib].
* OrangeFactor bug commenter task:
** brasstacks cron job run in both a daily and weekly variant, that adds failure summary comments to bugs associated with the intermittent test failures ([https://bugzilla.mozilla.org/show_bug.cgi?id=1365219#c6 example]).
** Fetches data from the OrangeFactor REST API.
** Posts bug comments using the orangefactor@bots.tld account, which has no additional permissions beyond a standard user account.
** [https://hg.mozilla.org/automation/orangefactor/file/default/woo_commenter.py Source].


=== Development & Deployment ===
* [https://activedata.allizom.org/ ActiveData] mirror of OrangeFactor Elasticsearch data:
** Currently synced manually by :eykle.
** Plans to automate this in the future ({{bug|1344253}}).


The Orange Factor web app can be run locally. See the instructions at:
== Deploying ==


https://hg.mozilla.org/automation/orangefactor/file/tip/README.txt
* Ensure you are a member of the LDAP group `vpn_brasstacks` and are on the sudoers list in puppet.
* Either connect to MozillaVPN or have the SSH jumphost set up via a `ProxyCommand` in your SSH config, then ssh to `brasstacks1.dmz.scl3.mozilla.com`.
* To pull new changes from the Hg repo:
sudo -u webtools hg pull -uv -R /home/webtools/apps/orangefactor/src/orangefactor/
* To then pick up any changes to the Python REST API (not required for changes to UI static assets or the Python scripts run by cron):
sudo service orangefactor stop; sudo service orangefactor start
* It's no longer necessary to sync the UI static assets in the source directory with the openresty/nginx default public root, since they are served directly from the source directory.


== Making Oranges Interesting ==
== Making Oranges Interesting ==
Line 71: Line 83:
* identify interesting patterns in failures; some failures may occur more frequently on certain OS's, build types, architectures, or other factors; by providing views which can track oranges across a range of factors, we might be able to provide developers with data that would help them reproduce failures or give them insight into their cause
* identify interesting patterns in failures; some failures may occur more frequently on certain OS's, build types, architectures, or other factors; by providing views which can track oranges across a range of factors, we might be able to provide developers with data that would help them reproduce failures or give them insight into their cause


* identify overall trends in orange occurrences, already part of [http://jmaher.couchone.com/orange_factor/_design/woo/orange.html Orange Factor]; this can help track the 'orangeness' of a product over time, and can help measure the helpfulness of orange-fixing activities
* identify overall trends in orange occurrences, already part of the [http://jmaher.couchone.com/orange_factor/_design/woo/orange.html legacy Orange Factor app]; this can help track the 'orangeness' of a product over time, and can help measure the helpfulness of orange-fixing activities


== Dashboard Views ==
== Attacking Oranges ==


A list of dashboard views that may be interesting. We're currently using Orange Factor as a platform to experiment with views.
https://developer.mozilla.org/en/QA/Fixing_intermittent_oranges


* {{done|display}} of overall orange factor over time
== History ==
* {{done|display}} of failures/day, for a given failure
* {{done|display}} of failures/commit/day, for a given failure
* {{done|display}} of moving averages of the above
* display of failure frequencies which exceed certain limits (probably based on standard deviation)
* {{done|display}} of most common failures, in aggregate, and separated by various factors: platform, OS version, architecture, build type, etc
* other...?


== Statistics ==
These projects are deprecated and replaced by the new War on Orange/OrangeFactor application.


The amount of information yielded from the parsed logs is vast.  The raw data will be noisy and the trends will not be easily discerned.  So statistical analysis should be used to manipulate the data and seek insight into trends.
'''Topfails'''


[[Auto-tools/Projects/WarOnOrange/Statistics|War On Orange Statistics]]
[http://brasstacks.mozilla.com/topfails/ Topfails] was the first database-driven orange tracker developed in our team.  It shows failures in terms of overall occurrences.  It suffers from a buggy log parser, and a UI with relatively few views.


== How Tinderbox Stores Its Data ==
source: http://hg.mozilla.org/automation/topfails/


Tinderbox stores logs in the format
'''Old Orange Factor'''


    1291054515.1291054887.26068.gz
[http://jmaher.couchone.com/orange_factor/_design/woo/orange.html Orange Factor] is a newer dashboard by jmaher.  It calculates the average number of oranges per push (the 'orange factor'), and tracks that number over time.  We're currently using it as a base to explore the usefulness of other statistics.


or
source: http://github.com/jmaher/Orange-Factor


    xxx.yyy.zzz.gz
See also [http://people.mozilla.com/~mcote/war-on-orange/war-on-orange-paper-testistanbul.pdf this paper] and the accompanying [http://people.mozilla.com/~mcote/war-on-orange/war-on-orange-testistanbul-slides-presentation.pdf slides], prepared for the TestIstanbul 2012 conference.


where
== ActiveData ==


    * xxx is approximately the time that buildbot picked up the test to run
The OrangeFactor ElasticSearch metadata is replicated to [https://activedata.allizom.org/ ActiveData], and can be queried there using the "orange_factor" index:
    * yyy is the time the log was e-mailed to tinderbox
    * zzz is the pid of the perl process that processed the log ([https://mxr.mozilla.org/webtools/source/tinderbox/processbuild.pl#143 no really])


Tinderbox maintains a list of bug->log associations at http://tinderbox.mozilla.org/Firefox/notes.txt.  The format used therein is:
: <code>{"from": "orange_factor"}</code>
 
    1291056044|WINNT 5.2 mozilla-central debug test mochitests-2/5|jmathies@mozilla.com|1291059392|Bug%20614474
 
or
 
    yyy|mmm|[user]|ttt|[bug #]
 
where
 
    * yyy is the same as yyy above
    * mmm is a string representing the testrun, in a format which isn't in the raw buildbot log
    * ttt is the time that the bug was starred
 
None of this data can be found in the raw buildbot logs themselves, although yyy is approximately the same as the timestamp of the logfile on stage.mozilla.org (they're not exact though, there is usually a few seconds difference between the time the log was e-mailed to tinderbox (yyy) and the time the log was copied to stage).
 
== ElasticSearch Queries ==
 
The log metadata is all stored in ElasticSearch, see the [[Auto-tools/Projects/WarOnOrange/ElasticSearch|ElasticSearch]] page for details on querying this database.
 
== Attacking Oranges ==
 
https://developer.mozilla.org/en/QA/Fixing_intermittent_oranges


== REST API ==
== REST API ==
Line 183: Line 165:


Returns the same data as the "bugs" property of the bybug returned data.
Returns the same data as the "bugs" property of the bybug returned data.
=== testrun ===
Returns information on one or more test runs.
Parameters:
The request can be made in one of two fashions. To get information about only one test run,
* starttime: Unix timestamp of the run's start time.
* machine: Hostname of the test machine.
Or to get information on several,
* runs: a comma-separated list of timestamps and machines, in the form <timestamp>|<machine>,<timestamp>|<machine>, eg      testrun?runs=1297070365|talos-r3-leopard-012,1295484366|talos-r3-xp-037
Returns an object with properties in the form '<timestamp>|<machine>' (regardless of which parameter format was used). Each property has an array of matching test runs, with these properties:
* tree
* passed/failed/todo: the number of tests in each category; may be missing for testruns that never completed due to crashes, etc
* elapsedtime: number of seconds it took for the testrun to complete
* suitename
* builder: the buildbot builder string
* machine: the machine name
* cmdline: the command line used to invoke the test
* buildtype: opt or debug
* platform: buildbot platform string
* date: date the testrun was run, YYYY-MM-DD
* buildid: the buildbot buildid used to run the test
* revision: the hg revision used to run the test
* testfailure_count: the number of failing tests in this testrun.
* testrunerrors: an array of errors that could not be pinned to a specific test; these are usually memory leaks or crashes that occur at the end of a testrun.  these errors are not included in 'testfailure_count'
* testfailures: an array of testfailures which is testfailure_count in length; each member of this array has two keys:
* test: the name of the test that failed
* failures: a list of failures that occurred
* logurl: URL to the complete test log.
=== testfailures ===
Returns information on test failures.
This is different from information on oranges, in that (a) these failures may not have been starred and (b) there may be more than one failure per test.
Parameters:
* startday: Mandatory. In ISO format, e.g. 2011-05-27.
* endday: Mandatory. Also in ISO format.
* tree: Optional, defaults to all.
* type: Build type, opt or debug, defaults to all.
Returns an object with properties named by test run ID and containing an array of failures.  Each failure is an object with the following properties:
* buildtype
* errors: an array of objects with 'status' and 'text' properties describing the error.
* testfailure_id
* buildid
* os
* testgroup_id
* tree
* machine
* platform
* test
* starttime
* date
* duration
* testgroup
* revision
* testsuite_id
* logurl
Edmorley
canmove, Confirmed users
1,126

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1027575...1195317"