TestEngineering/Performance/Results: Difference between revisions

← Older edit

TestEngineering/Performance/Results (view source)

Revision as of 16:17, 11 August 2021

1,037 bytes removed , 11 August 2021

→‎What do we have?

Davehunt

Confirmed users

2,177

edits

@@ Line 1: / Line 1: @@
-{{DISPLAYTITLE:Performance Results}}
+{{DISPLAYTITLE:Firefox Performance Results}}
+There are a variety of places where performance results are available, some of these can be queried directly, and others are dashboards built on top of those sources.
-There are a variety of places where performance results are available, some of these can be queried directly, and others are dashboards built on top of those sources.
+= What should I use? =
+{{todo|for now just gathering ideas}}
+* Does my patch affect performance?
+** [[#Compare|Perfherder compare]]
+* Did my push affect performance?
+** [[#Compare|Perfherder compare]]
+* How does Firefox compare against other browsers?
+** [[#Firefox Performance Dashboard|Firefox Performance Dashboard]]
+* Are we meeting our goals?
+* What performance tests are we running?
+* What platforms are we running performance tests against?
-= ActiveData Query Tool =
+= What do we have? =
-ActiveData is a collection of billions of records covering unit tests, jobs, performance data, and source code revision history. The intended use is for automated clients or for building dashboards, however the query tool can be used for exploration.
-* Location: https://activedata.allizom.org/tools/query.html
+== Firefox Performance Dashboard ==
+* Location: https://arewefastyet.com/
 * Access: Public
-* Audience: Engineers
+* Audience: Managers, Engineers
-* Documentation: [[EngineeringProductivity/Projects/ActiveData]]
+* Documentation: {{todo|add link to documentation}}
-* Source code: https://github.com/mozilla/activedata
+* Source code: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard
-* Issue tracker: https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=ActiveData
+* Issue tracker: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard/issues
-* Contact: Kyle Lahnakoski [ekyle]
+* Contact: Dave Hunt [davehunt]
-== Examples ==
+== Metrics ==
-* [https://activedata.allizom.org/tools/query.html#query_id=rqnxuVpf Performance frameworks]
+{{todo|https://metrics.mozilla.com/protected/shiny/cdowhygelund/firefox-perf-telemetry-dashboard/}}
-* [https://activedata.allizom.org/tools/query.html#query_id=rqnxuVpf Performance suites]
-* [https://activedata.allizom.org/tools/query.html#query_id=1dtPBeZR Performance suites running on Android hardware]
-= ActiveData Recipes =
+== Perfherder ==
-Due to the size and complexity of ActiveData, using it to answer everyday questions is not a straightforward task and the built-in query tool is often insufficient. ActiveData Recipes was created to provide an intuitive way to not only use ActiveData, but to save your queries for future use and share them with others.
+Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products. Perfherder is part of the [[EngineeringProductivity/Projects/Treeherder|Treeherder]] project.
-* Location: https://adr-dev.herokuapp.com/
+* Location: https://treeherder.mozilla.org/perf.html
 * Access: Public
-* Audience: Engineers, Managers
+* Audience: Sheriffs, Engineers
-* Documentation: https://active-data-recipes.readthedocs.io/
+* Documentation: [[EngineeringProductivity/Projects/Perfherder]]
-* Source code: https://github.com/mozilla/active-data-recipes
+* Source code: https://github.com/mozilla/treeherder
-* Issue tracker: https://github.com/mozilla/active-data-recipes/issues
+* Issue tracker: https://bugzilla.mozilla.org/buglist.cgi?product=Tree%20Management&component=Perfherder&resolution=---
-* Contact: Andrew Halberstadt [ahal]
+* Contact: Sarah Clements [sclements]
-= Datadog =
+=== Alerts ===
 {{todo|add description}}
-== WebPageTest ==
+* Location: https://treeherder.mozilla.org/perf.html#/alerts
-* Location: https://app.datadoghq.com/dashboard/lists/manual/18187
+* Documentation: {{todo|}}
-* Access: File a bug to request access by replacing '''[user]''' and '''[email]''' in [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=normal&bug_status=NEW&bug_type=task&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox68=---&cf_status_firefox69=---&cf_status_firefox70=---&cf_status_firefox_esr60=---&cf_status_firefox_esr68=---&cf_tracking_firefox68=---&cf_tracking_firefox69=---&cf_tracking_firefox70=---&cf_tracking_firefox_esr60=---&cf_tracking_firefox_esr68=---&cf_tracking_firefox_relnote=---&comment=Please%20add%20%5Bemail%5D%20to%20the%20cloudservices_datadog%20LDAP%20group%20and%20be%20sent%20an%20invitation%20to%20access%20Datadog%3F&component=Operations&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-607=X&flag_type-708=X&flag_type-721=X&flag_type-737=X&flag_type-748=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-929=X&flag_type-936=X&flag_type-941=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=Grant%20%5Buser%5D%20access%20to%20Datadog&target_milestone=---&version=unspecified this template].
-* Audience: Engineers, Managers
-* Documentation: https://mozilla-wpt-api-docs.readthedocs.io/ {{todo|add doc on how to use datadog}}
-* Source code: https://github.com/mozilla/wpt-api
-* Issue tracker: https://github.com/mozilla/wpt-api/issues
-* Contact: Stephen Donner [stephend]
-= Firefox Health =
+=== Compare ===
-The health dashboard tracks metrics and statistics important for tracking performance improvements.
+{{todo|add description}}
-* Location: https://health.graphics/
+* Location: https://treeherder.mozilla.org/perf.html#/comparechooser
-* Access: Public
+* Documentation: {{todo|}}
-* Audience: Managers
-* Documentation: {{todo|add link to documentation}}
+=== Graphs ===
-* Source code: https://github.com/mozilla-frontend-infra/firefox-health-dashboard
+{{todo|add description}}
-* Issue tracker: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues
-* Contact: Kyle Lahnakoski [ekyle]
+* Location: https://treeherder.mozilla.org/perf.html#/graphs
+* Documentation: {{todo|}}
-= Firefox Performance Dashboard =
+== STMO ==
-* Location: https://arewefastyet.com/
+STMO is shorthand for [https://sql.telemetry.mozilla.org/ sql.telemetry.mozilla.org], an installation of the excellent [https://redash.io/ Re:dash] data analysis and dashboarding tool that has been customized and configured for use with a number of the Firefox organisation's data sets.
-* Access: Public
-* Audience: Managers
-* Documentation: {{todo|add link to documentation}}
-* Source code: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard
-* Issue tracker: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard/issues
-* Contact: Armen Zambrano Gasparnian [armenzg]
-= Perfherder =
+* Location: https://sql.telemetry.mozilla.org/
-Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products. Perfherder is part of the [[EngineeringProductivity/Projects/Treeherder|Treeherder]] project.
+* Access: LDAP via SSO
+* Audience: Sheriffs, Engineers, Managers
+* Documentation: https://docs.telemetry.mozilla.org/tools/stmo.html
+* Source code: https://github.com/mozilla/redash
+* Issue tracker: https://github.com/mozilla/redash/issues
+* Contact: #fx-metrics on Slack
-= Redash =
+== Ad-hoc results ==
+Before performance tests are fully automated and integrated into our pipeline, we often run manual or semi-automated tests. These results are often presented in ways that don't fit into our existing reporting structures, such as spreadsheets or documents. The following list of links serves as a repository for these results:
+* [https://drive.google.com/drive/u/0/folders/1sZ6C6bku2vR575GSH5DxgSlP43JYT-Je Performance/Results]
+* [https://drive.google.com/drive/folders/11a90GH_xwcqwkcAmzCZAkwmQeuSoXA3n Performance/Projects/Desktop/Testing]
-= Ad-hoc results =
+Please make sure when you add results to the above locations that the name or internal labeling is clear as to what was tested, the methods used, when the results were gathered, and anything else that may be useful.