canmove, Confirmed users
1,126
edits
Auto-tools/Projects/OrangeFactor: Difference between revisions
m
Make decommission note larger
m (Make decommission note larger) |
|||
| (28 intermediate revisions by 6 users not shown) | |||
| Line 1: | Line 1: | ||
== | = OrangeFactor has been decommissioned = | ||
'''It has been succeeded by [[EngineeringProductivity/Projects/IntermittentFailuresView|Intermittent Failures View]].''' | |||
== About == | |||
The [https://brasstacks.mozilla.com/orangefactor/ OrangeFactor] web app is a tool for tracking and analysing intermittent test failures encountered that occur during Firefox/gecko continuous integration automation. | |||
NB: OrangeFactor is considered near end of life. A replacement will likely use Treeherder's API as a backend instead. | |||
== | == Contributing == | ||
For help with OrangeFactor, please contact #ateam or :emorley, :gbrown, :jmaher. | |||
File bugs under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=OrangeFactor Tree Management::OrangeFactor]. See open bugs [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&component=OrangeFactor&product=Tree+Management here] | |||
The OrangeFactor web app can be run locally. See the instructions at: | |||
https://hg.mozilla.org/automation/orangefactor/file/default/README.txt | |||
== Architecture == | |||
== | |||
* [https://brasstacks.mozilla.com/ brasstacks.mozilla.com] | |||
** SSL termination occurs on the box. | |||
** Only listens on port 443, since HTTP->HTTPS redirection performed by Zeus. | |||
[ | * [https://brasstacks.mozilla.com/orangefactor/ OrangeFactor UI]: | ||
** Static UI that interacts with the OrangeFactor API. | |||
** Served by [https://openresty.org/ OpenResty]. | |||
** [https://groups.google.com/forum/#!topic/mozilla.dev.tree-management/cIc63kVSN5M Requires LDAP credentials to access]. | |||
** [https://hg.mozilla.org/automation/orangefactor/file/default/html Source]. | |||
* [https://brasstacks.mozilla.com/orangefactor/api/ OrangeFactor REST API]: | |||
** Python FastCGI app reverse proxied by OpenResty. | |||
** [https://groups.google.com/forum/#!topic/mozilla.dev.tree-management/cIc63kVSN5M Requires LDAP credentials to access]. | |||
** Read-only API apart from job classification submissions from [https://github.com/mozilla/treeherder Treeherder], which use [https://github.com/kumar303/mohawk Hawk authentication]. | |||
** API responses are a combination of results from OrangeFactor's ES instance and the public hg.mozilla.org pushlog. | |||
** ES queries are made using [https://github.com/aparo/pyes pyes], plus a helper library we've written on top of it, [https://hg.mozilla.org/automation/mozautoeslib/ mozautoeslib]. | |||
** [https://hg.mozilla.org/automation/orangefactor/file/default/server Source]. | |||
* OrangeFactor Elasticsearch instance: | |||
** Index `bugs`: The intermittent test failure records submitted by Treeherder via OrangeFactor's REST API. | |||
** Index `bzcache`: A cache of public `keyword:intermittent-failure` bugs populated via a brasstacks cron job. | |||
* OrangeFactor bzcache refresh task: | |||
** brasstacks cron job run every four hours, that populates the `bzcache` index on the OrangeFactor Elasticsearch instance. | |||
** Fetches bug data using unauthenticated requests to Bugzilla's REST API. | |||
** [https://github.com/jonallengriffin/bzcache/blob/master/bzcache/bz_cache_refresh.py Source]. | |||
* | * OrangeFactor mailer task: | ||
** brasstacks cron job <del>run every week</del> now disabled ({{bug|1426416}}), that emails a summary of top failures to [https://groups.google.com/forum/#!forum/mozilla.dev.tree-alerts mozilla.dev.tree-alerts]. | |||
** Fetches data from the OrangeFactor REST API. | |||
** [https://hg.mozilla.org/automation/orangefactor/file/default/woo_mailer.py Source]. | |||
* a [ | * OrangeFactor bug commenter task: | ||
** brasstacks cron job run in both a daily and weekly variant, that adds failure summary comments to bugs associated with the intermittent test failures ([https://bugzilla.mozilla.org/show_bug.cgi?id=1365219#c6 example]). | |||
** Fetches data from the OrangeFactor REST API. | |||
** Posts bug comments using the orangefactor@bots.tld account, which has no additional permissions beyond a standard user account. | |||
** [https://hg.mozilla.org/automation/orangefactor/file/default/woo_commenter.py Source]. | |||
* | * [https://activedata.allizom.org/ ActiveData] mirror of OrangeFactor Elasticsearch data: | ||
** Currently synced manually by :eykle. | |||
** Plans to automate this in the future ({{bug|1344253}}). | |||
== Deploying == | |||
* a | * Ensure you are a member of the LDAP group `vpn_brasstacks` and are on the sudoers list in puppet. | ||
* Either connect to MozillaVPN or have the SSH jumphost set up via a `ProxyCommand` in your SSH config, then ssh to `brasstacks1.dmz.scl3.mozilla.com`. | |||
* To pull new changes from the Hg repo: | |||
sudo -u webtools hg pull -uv -R /home/webtools/apps/orangefactor/src/orangefactor/ | |||
* To then pick up any changes to the Python REST API (not required for changes to UI static assets or the Python scripts run by cron): | |||
sudo service orangefactor stop; sudo service orangefactor start | |||
* It's no longer necessary to sync the UI static assets in the source directory with the openresty/nginx default public root, since they are served directly from the source directory. | |||
== Making Oranges Interesting == | == Making Oranges Interesting == | ||
| Line 65: | Line 83: | ||
* identify interesting patterns in failures; some failures may occur more frequently on certain OS's, build types, architectures, or other factors; by providing views which can track oranges across a range of factors, we might be able to provide developers with data that would help them reproduce failures or give them insight into their cause | * identify interesting patterns in failures; some failures may occur more frequently on certain OS's, build types, architectures, or other factors; by providing views which can track oranges across a range of factors, we might be able to provide developers with data that would help them reproduce failures or give them insight into their cause | ||
* identify overall trends in orange occurrences, already part of [http://jmaher.couchone.com/orange_factor/_design/woo/orange.html Orange Factor]; this can help track the 'orangeness' of a product over time, and can help measure the helpfulness of orange-fixing activities | * identify overall trends in orange occurrences, already part of the [http://jmaher.couchone.com/orange_factor/_design/woo/orange.html legacy Orange Factor app]; this can help track the 'orangeness' of a product over time, and can help measure the helpfulness of orange-fixing activities | ||
== Attacking Oranges == | |||
https://developer.mozilla.org/en/QA/Fixing_intermittent_oranges | |||
== History == | |||
These projects are deprecated and replaced by the new War on Orange/OrangeFactor application. | |||
'''Topfails''' | |||
[http://brasstacks.mozilla.com/topfails/ Topfails] was the first database-driven orange tracker developed in our team. It shows failures in terms of overall occurrences. It suffers from a buggy log parser, and a UI with relatively few views. | |||
source: http://hg.mozilla.org/automation/topfails/ | |||
'''Old Orange Factor''' | |||
[http://jmaher.couchone.com/orange_factor/_design/woo/orange.html Orange Factor] is a newer dashboard by jmaher. It calculates the average number of oranges per push (the 'orange factor'), and tracks that number over time. We're currently using it as a base to explore the usefulness of other statistics. | |||
source: http://github.com/jmaher/Orange-Factor | |||
[[ | See also [http://people.mozilla.com/~mcote/war-on-orange/war-on-orange-paper-testistanbul.pdf this paper] and the accompanying [http://people.mozilla.com/~mcote/war-on-orange/war-on-orange-testistanbul-slides-presentation.pdf slides], prepared for the TestIstanbul 2012 conference. | ||
== | == ActiveData == | ||
The OrangeFactor ElasticSearch metadata is replicated to [https://activedata.allizom.org/ ActiveData], and can be queried there using the "orange_factor" index: | |||
: <code>{"from": "orange_factor"}</code> | |||
== REST API == | |||
The War on Orange site pulls its data from a REST API. Other applications can hook into this to get the raw orange data. | |||
The API root is at http://brasstacks.mozilla.com/orangefactor/api/. Parameters are passed via the query string, eg. ?key1=value1&key2=value2. Example: http://brasstacks.mozilla.com/orangefactor/api/count?startday=2011-05-21&endday=2011-05-27&tree=mozilla-central | |||
All returned data is in JSON format. | |||
=== bybug === | |||
Provides a date-indexed list of oranges, with bug numbers, along with minimal details of each bug. | |||
or | Parameters: | ||
* startday: Mandatory. In ISO format, e.g. 2011-05-27. | |||
* endday: Mandatory. Also in ISO format. | |||
* bugid: Optional. Return orange data for this bug only. | |||
* tree: Optional. Return information about this tree only. Defaults to mozilla-central. Pass "all" for orange data on all trees. | |||
* type: Optional. Return information for this build type only. Must be "opt" or "debug". Defaults to none (both build types). | |||
Returns an object with two properties: | |||
* oranges: An object with dates as properties, e.g. data['oranges']['2011-05-27']. Each property is another object with orange data for the day, with the following properties: | |||
** orangecount: total number of oranges for that day, e.g. 54. | |||
** testruns: number of test runs that day, e.g. 24. The "Orange Factor" is orangecount/testruns. | |||
** oranges: details of the oranges that occurred that day. It is an array of objects, each one having these simple properties: | |||
*** buildtype | |||
*** date | |||
*** starttime | |||
*** timestamp | |||
*** platform | |||
*** branch | |||
*** machinename | |||
*** test | |||
*** bug | |||
*** revision | |||
* bugs: An object with bug ids as properties. Each bug in the above list of oranges is represented here. The information is gathered via pulse and thus is quicker to access than querying Bugzilla. Only a few basic properties are available; for more detailed info, you will have to consult Bugzilla: | |||
** id | |||
** status | |||
** summary | |||
=== count === | |||
Returns a date-indexed summary of orange data. | |||
The parameters are the same as for bybug. The returned data is also the same, except that the 'bugs' property is not returned, and the array of orange details (data['oranges'][<date>]['oranges']) is empty. This is a faster way to get just the summarized numerical data for, e.g., Orange Factor calculations. | |||
== | === bugdetails === | ||
Returns minimal details for one or more bugs. | |||
The only parameter is "bugid", which takes a bug id or a comma-separated list of bug ids. | |||
Returns the same data as the "bugs" property of the bybug returned data. | |||