TestEngineering/Performance

Objectives

• Increase regression detection coverage to page load on Android products including WebView comparisons
• Increase quality standards by measuring and alerting on power usage for Android and ARM64
• Build dashboards to improve visibility of how we are tracking against our release criteria for Android
• Reducing noise in test results and adding annotations to provide clarity of signal for regressions
• Develop measurements and mechanisms for reporting on our tools, policies and documentation for how to improve clarity and efficiency of risk assessments

Team

• Dave Hunt [:davehunt]
• Rob Wood [:rwood]
• Kyle Lahnakoski [:ekyle]
• Stephen Donner [:stephend]
• Ionut Goldan [:igoldan]
• Florian Strugariu [:bebe]
• Greg Mierzwinski [:sparky]

Projects

Trello: https://trello.com/b/gc3AlyhB/firefox-performance-test-engineering

Raptor

Raptor is a Python testing framework used to run browser benchmark and browser page-load performance tests. Raptor is cross-browser compatible and is currently running in Mozilla taskcluster production on Firefox Desktop, Firefox Android, and on Google Chromium.

• Owner: Rob Wood [:rwood]
• Source: https://searchfox.org/mozilla-central/source/testing/raptor
• Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DRaptor
• User documentation: https://wiki.mozilla.org/Performance_sheriffing/Raptor
• Developer documentation: https://wiki.mozilla.org/Performance_sheriffing/Raptor

Talos

Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.

• Owner: Rob Wood [:rwood], Joel Maher [:jmaher]
• Source: https://searchfox.org/mozilla-central/source/testing/talos
• Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DTalos
• User documentation: https://wiki.mozilla.org/Performance_sheriffing/Talos
• Developer documentation: https://wiki.mozilla.org/Performance_sheriffing/Talos

WebPageTest

• Owner: Stephen Donner [:stephend]
• Source: https://github.com/mozilla/wpt-api
• (User and Dev) Docs: https://mozilla-wpt-api-docs.readthedocs.io/en/master/
• "help-wanted" labelled Issues: https://github.com/mozilla/wpt-api/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22

Perfherder

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android). Perfherder is part of the Treeherder project. The production version of it lives at https://treeherder.mozilla.org/perf.html

Firefox Are We Fast Yet Dashboard

Shows a variety of benchmarks, run on a variety of platforms. Meant to be a detailed view of performance.

• Location: https://arewefastyet.com
• Owner: Armen Zambrano Gasparnian [:armen]
• Source: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard

Firefox Health Dashboard

The health dashboard tracks metrics and statistics important for tracking performance improvements. Meant to be a high level view.

• Location: https://health.graphics/
• Owner: Armen Zambrano Gasparnian [:armen], Kyle Lahnakoski [:ekyle]
• Source: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/
• Good first bugs: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22