TestEngineering/Performance

Objectives

• Increase regression detection coverage to page load on Android products including WebView comparisons
• Increase quality standards by measuring and alerting on power usage for Android and ARM64
• Build dashboards to improve visibility of how we are tracking against our release criteria for Android
• Reducing noise in test results and adding annotations to provide clarity of signal for regressions
• Develop measurements and mechanisms for reporting on our tools, policies and documentation for how to improve clarity and efficiency of risk assessments

Team

• Dave Hunt [:davehunt]
• Rob Wood [:rwood]
• Kyle Lahnakoski [:ekyle]
• Stephen Donner [:stephend]
• Ionut Goldan [:igoldan]
• Florian Strugariu [:bebe]
• Greg Mierzwinski [:sparky]

Projects

Trello: https://trello.com/b/gc3AlyhB/firefox-performance-test-engineering

Raptor

Raptor is a Python testing framework used to run browser benchmark and browser page-load performance tests. Raptor is cross-browser compatible and is currently running in Mozilla taskcluster production on Firefox Desktop, Firefox Android, and on Google Chromium.

• Owner: Rob Wood [:rwood]
• Source: https://searchfox.org/mozilla-central/source/testing/raptor
• Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DRaptor
• User documentation: https://wiki.mozilla.org/Performance_sheriffing/Raptor
• Developer documentation: https://wiki.mozilla.org/Performance_sheriffing/Raptor

Talos

Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.

• Owner: Rob Wood [:rwood], Joel Maher [:jmaher]
• Source: https://searchfox.org/mozilla-central/source/testing/talos
• Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DTalos
• User documentation: https://wiki.mozilla.org/Performance_sheriffing/Talos
• Developer documentation: https://wiki.mozilla.org/Performance_sheriffing/Talos

WebPageTest

• Owner: Stephen Donner [:stephend]
• Source: https://github.com/mozilla/wpt-api
• (User and Dev) Docs: https://mozilla-wpt-api-docs.readthedocs.io/en/master/
• "help-wanted" labelled Issues: https://github.com/mozilla/wpt-api/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22

Firefox Are We Fast Yet Dashboard

Shows a variety of benchmarks, run on a variety of platforms. Meant to be a detailed view of performance.

   Location: https://arewefastyet.com
   Owner: Armen Zambrano Gasparnian [:armen]
   Source: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard

Firefox Health Dashboard

The health dashboard tracks metrics and statistics important for tracking performance improvements. Meant to be a high level view.

   Location: https://health.graphics/
   Owner: Armen Zambrano Gasparnian [:armen], Kyle Lahnakoski [:ekyle]
   Source: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/
   Good first bugs: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22