TestEngineering/Performance/Results

There are a variety of places where performance results are available, some of these can be queried directly, and others are dashboards built on top of those sources.

What should I use?

[TODO] for now just gathering ideas

• Does my patch affect performance?
• Did my push affect performance?
• How does Firefox compare against other browsers?
• Are we meeting our goals?
• What performance tests are we running?
• What platforms are we running performance tests against?

What do we have?

ActiveData Query Tool

ActiveData is a collection of billions of records covering unit tests, jobs, performance data, and source code revision history. The intended use is for automated clients or for building dashboards, however the query tool can be used for exploration.

• Location: https://activedata.allizom.org/tools/query.html
• Access: Public
• Audience: Engineers
• Documentation: EngineeringProductivity/Projects/ActiveData
• Source code: https://github.com/mozilla/activedata
• Issue tracker: https://bugzilla.mozilla.org/buglist.cgi?product=Testing&component=ActiveData&resolution=---
• Contact: Kyle Lahnakoski [ekyle]

Examples

• Performance frameworks
• Performance suites
• Performance suites running on Android hardware

ActiveData Recipes

Due to the size and complexity of ActiveData, using it to answer everyday questions is not a straightforward task and the built-in query tool is often insufficient. ActiveData Recipes was created to provide an intuitive way to not only use ActiveData, but to save your queries for future use and share them with others.

• Location: https://adr-dev.herokuapp.com/
• Access: Public
• Audience: Engineers, Managers
• Documentation: https://active-data-recipes.readthedocs.io/
• Source code: https://github.com/mozilla/active-data-recipes
• Issue tracker: https://github.com/mozilla/active-data-recipes/issues
• Contact: Andrew Halberstadt [ahal]

Datadog

[TODO] add description

WebPageTest

• Location: https://app.datadoghq.com/dashboard/lists/manual/18187
• Access: File a bug to request access by replacing [user] and [email] in this template.
• Audience: Engineers, Managers
• Documentation: https://mozilla-wpt-api-docs.readthedocs.io/
  [TODO] add doc on how to use datadog
• Source code: https://github.com/mozilla/wpt-api
• Issue tracker: https://github.com/mozilla/wpt-api/issues
• Contact: Stephen Donner [stephend]

Firefox Health

The health dashboard tracks metrics and statistics important for tracking performance improvements.

• Location: https://health.graphics/
• Access: Public
• Audience: Managers
• Documentation:
  [TODO] add link to documentation
• Source code: https://github.com/mozilla-frontend-infra/firefox-health-dashboard
• Issue tracker: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues
• Contact: Kyle Lahnakoski [ekyle]

Firefox Performance Dashboard

• Location: https://arewefastyet.com/
• Access: Public
• Audience: Managers
• Documentation:
  [TODO] add link to documentation
• Source code: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard
• Issue tracker: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard/issues
• Contact: Armen Zambrano Gasparnian [armenzg]

Metrics

[TODO] https://metrics.mozilla.com/protected/shiny/cdowhygelund/firefox-perf-telemetry-dashboard/

Nimbledroid

[TODO] add description

• Location: https://nimbledroid.com/my_apps
• Access: Private
  [TODO] add details
• Audience: Engineers
• Documentation:
  [TODO] add link to documentation
• Issue tracker:
  [TODO]
• Contact: No-Jun Park [npark]

Perfherder

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products. Perfherder is part of the Treeherder project.

• Location: https://treeherder.mozilla.org/perf.html
• Access: Public
• Audience: Sheriffs, Engineers
• Documentation: EngineeringProductivity/Projects/Perfherder
• Source code: https://github.com/mozilla/treeherder
• Issue tracker: https://bugzilla.mozilla.org/buglist.cgi?product=Tree%20Management&component=Perfherder&resolution=---
• Contact: Sarah Clements [sclements]

Alerts

[TODO] add description

• Location: https://treeherder.mozilla.org/perf.html#/alerts
• Documentation:
  [TODO]

Compare

[TODO] add description

• Location: https://treeherder.mozilla.org/perf.html#/comparechooser
• Documentation:
  [TODO]

Graphs

[TODO] add description

• Location: https://treeherder.mozilla.org/perf.html#/graphs
• Documentation:
  [TODO]

Redash

[TODO] add description

• Location: https://sql.telemetry.mozilla.org/
• Access:
  [TODO]
• Audience: Sheriffs, Engineers, Managers
• Documentation:
  [TODO]
• Source code:
  [TODO]
• Issue tracker:
  [TODO]
• Contact:
  [TODO]

Ad-hoc results

[TODO] add description

• Performance/Projects/Desktop/Testing