Auto-tools/Projects/OrangeFactor

OrangeFactor has been decommissioned

It has been succeeded by Intermittent Failures View.

About

The OrangeFactor web app is a tool for tracking and analysing intermittent test failures encountered that occur during Firefox/gecko continuous integration automation.

NB: OrangeFactor is considered near end of life. A replacement will likely use Treeherder's API as a backend instead.

Contributing

For help with OrangeFactor, please contact #ateam or :emorley, :gbrown, :jmaher.

File bugs under Tree Management::OrangeFactor. See open bugs here

The OrangeFactor web app can be run locally. See the instructions at: https://hg.mozilla.org/automation/orangefactor/file/default/README.txt

Architecture

• brasstacks.mozilla.com
  • SSL termination occurs on the box.
  • Only listens on port 443, since HTTP->HTTPS redirection performed by Zeus.

• OrangeFactor UI:
  • Static UI that interacts with the OrangeFactor API.
  • Served by OpenResty.
  • Requires LDAP credentials to access.
  • Source.

• OrangeFactor REST API:
  • Python FastCGI app reverse proxied by OpenResty.
  • Requires LDAP credentials to access.
  • Read-only API apart from job classification submissions from Treeherder, which use Hawk authentication.
  • API responses are a combination of results from OrangeFactor's ES instance and the public hg.mozilla.org pushlog.
  • ES queries are made using pyes, plus a helper library we've written on top of it, mozautoeslib.
  • Source.

• OrangeFactor Elasticsearch instance:
  • Index `bugs`: The intermittent test failure records submitted by Treeherder via OrangeFactor's REST API.
  • Index `bzcache`: A cache of public `keyword:intermittent-failure` bugs populated via a brasstacks cron job.

• OrangeFactor bzcache refresh task:
  • brasstacks cron job run every four hours, that populates the `bzcache` index on the OrangeFactor Elasticsearch instance.
  • Fetches bug data using unauthenticated requests to Bugzilla's REST API.
  • Source.

• OrangeFactor mailer task:
  • brasstacks cron job run every week now disabled (bug 1426416), that emails a summary of top failures to mozilla.dev.tree-alerts.
  • Fetches data from the OrangeFactor REST API.
  • Source.

• OrangeFactor bug commenter task:
  • brasstacks cron job run in both a daily and weekly variant, that adds failure summary comments to bugs associated with the intermittent test failures (example).
  • Fetches data from the OrangeFactor REST API.
  • Posts bug comments using the orangefactor@bots.tld account, which has no additional permissions beyond a standard user account.
  • Source.

• ActiveData mirror of OrangeFactor Elasticsearch data:
  • Currently synced manually by :eykle.
  • Plans to automate this in the future (bug 1344253).

Deploying

• Ensure you are a member of the LDAP group `vpn_brasstacks` and are on the sudoers list in puppet.
• Either connect to MozillaVPN or have the SSH jumphost set up via a `ProxyCommand` in your SSH config, then ssh to `brasstacks1.dmz.scl3.mozilla.com`.
• To pull new changes from the Hg repo:

sudo -u webtools hg pull -uv -R /home/webtools/apps/orangefactor/src/orangefactor/

• To then pick up any changes to the Python REST API (not required for changes to UI static assets or the Python scripts run by cron):

sudo service orangefactor stop; sudo service orangefactor start

• It's no longer necessary to sync the UI static assets in the source directory with the openresty/nginx default public root, since they are served directly from the source directory.

Making Oranges Interesting

Currently, our intermittent oranges are not very interesting. After they've been identified, they are usually more-or-less ignored. This has caused us to accumulate oranges to the point where we have to deal with several of them for every commit (and by 'deal with', I mean 'log it and forget it'), which is time consuming for the sheriffs and for anyone who pushes a commit. At the same time, it demotivates any effort to actually fix them.

We'd like to help change that. We think we can help by creating a dashboard to analyze oranges in the following ways:

• identify the oranges that occur most frequently; these are the oranges that would produce the greatest improvement in our orange factor if fixed

• identify significant changes in the frequency of a given orange; if a known intermittent orange suddenly begins to occur more frequently, it may be related to a recent code change, and this might give developers more information about when/why it occurred, which would hopefully help in fixing it

• identify interesting patterns in failures; some failures may occur more frequently on certain OS's, build types, architectures, or other factors; by providing views which can track oranges across a range of factors, we might be able to provide developers with data that would help them reproduce failures or give them insight into their cause

• identify overall trends in orange occurrences, already part of the legacy Orange Factor app; this can help track the 'orangeness' of a product over time, and can help measure the helpfulness of orange-fixing activities

Attacking Oranges

https://developer.mozilla.org/en/QA/Fixing_intermittent_oranges

History

These projects are deprecated and replaced by the new War on Orange/OrangeFactor application.

Topfails

Topfails was the first database-driven orange tracker developed in our team. It shows failures in terms of overall occurrences. It suffers from a buggy log parser, and a UI with relatively few views.

source: http://hg.mozilla.org/automation/topfails/

Old Orange Factor

Orange Factor is a newer dashboard by jmaher. It calculates the average number of oranges per push (the 'orange factor'), and tracks that number over time. We're currently using it as a base to explore the usefulness of other statistics.

source: http://github.com/jmaher/Orange-Factor

See also this paper and the accompanying slides, prepared for the TestIstanbul 2012 conference.

ActiveData

The OrangeFactor ElasticSearch metadata is replicated to ActiveData, and can be queried there using the "orange_factor" index:

{"from": "orange_factor"}

REST API

The War on Orange site pulls its data from a REST API. Other applications can hook into this to get the raw orange data.

The API root is at http://brasstacks.mozilla.com/orangefactor/api/. Parameters are passed via the query string, eg. ?key1=value1&key2=value2. Example: http://brasstacks.mozilla.com/orangefactor/api/count?startday=2011-05-21&endday=2011-05-27&tree=mozilla-central

All returned data is in JSON format.

bybug

Provides a date-indexed list of oranges, with bug numbers, along with minimal details of each bug.

Parameters:

• startday: Mandatory. In ISO format, e.g. 2011-05-27.
• endday: Mandatory. Also in ISO format.
• bugid: Optional. Return orange data for this bug only.
• tree: Optional. Return information about this tree only. Defaults to mozilla-central. Pass "all" for orange data on all trees.
• type: Optional. Return information for this build type only. Must be "opt" or "debug". Defaults to none (both build types).

Returns an object with two properties:

• oranges: An object with dates as properties, e.g. data['oranges']['2011-05-27']. Each property is another object with orange data for the day, with the following properties:
  • orangecount: total number of oranges for that day, e.g. 54.
  • testruns: number of test runs that day, e.g. 24. The "Orange Factor" is orangecount/testruns.
  • oranges: details of the oranges that occurred that day. It is an array of objects, each one having these simple properties:
    • buildtype
    • date
    • starttime
    • timestamp
    • platform
    • branch
    • machinename
    • test
    • bug
    • revision
• bugs: An object with bug ids as properties. Each bug in the above list of oranges is represented here. The information is gathered via pulse and thus is quicker to access than querying Bugzilla. Only a few basic properties are available; for more detailed info, you will have to consult Bugzilla:
  • id
  • status
  • summary

count

Returns a date-indexed summary of orange data.

The parameters are the same as for bybug. The returned data is also the same, except that the 'bugs' property is not returned, and the array of orange details (data['oranges'][<date>]['oranges']) is empty. This is a faster way to get just the summarized numerical data for, e.g., Orange Factor calculations.

bugdetails

Returns minimal details for one or more bugs.

The only parameter is "bugid", which takes a bug id or a comma-separated list of bug ids.

Returns the same data as the "bugs" property of the bybug returned data.