TestEngineering/Performance: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Redirected page to Performance/Tools)
 
(38 intermediate revisions by 7 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Firefox Performance Test Engineering 🔥🦊⏱}}
#REDIRECT [[Performance/Tools]]


= Who we are =
{{DISPLAYTITLE:Firefox Performance Test Engineering 🔥🦊⏱️}}[[File:Fxperftest.png|thumb|right]]
* Dave Hunt [:davehunt] 🇬🇧
 
* Stephen Donner [:stephend] 🇺🇸
= New contributors =
* Rob Wood [:rwood] 🇨🇦
 
* Greg Mierzwinski [:sparky] 🇨🇦
If you are a new contributor, or would like to start contributing you can find a guide to help you [[/NewContributors|here]].
* Ionuț Goldan [:igoldan] 🇷🇴
* Florian Strugariu [:bebe] 🇷🇴
* Marian Raiciof [:marauder] 🇷🇴
* Alex Ionescu [:alexandrui] 🇷🇴
* Alex Irimovici [:air] 🇷🇴


= Where to find us =
= Where to find us =
* [[IRC]]: #perftest
* [https://chat.mozilla.org/#/room/#perftest:mozilla.org #perftest]
* Slack: #perftest
 
= Team purpose =
To support the infrastructure and creation of automated tests for evaluating the performance of Firefox products. This provides value by exposing gaps in coverage, revealing areas where we can make performance gains, identifying performance regressions in a timely manner, and by providing key performance metrics that assist in determining how Firefox measures against release criteria.


= What we do =
= What we do =
We provide automated testing support for measuring the performance of Firefox products. Here are a few examples of what you can expect our team to be working on.
* Identification of gaps in performance test infrastructure and monitoring.
* Provide advice/troubleshooting related to performance testing or any of our tools and harnesses.
* Designing and building performance test infrastructure and monitoring solutions.
* Develop test plans that involve automation of performance testing.
* Supporting Firefox engineers on writing performance tests.
* Prototype, build, and maintain test harnesses for performance testing.
* Supporting Firefox engineers on investigating regressions identified by tests.
* Monitoring and reporting of performance regressions.
* Collaboration with release operations on infrastructure requirements.
* Standing up performance tests in continuous integration environments.
* Monitoring performance test results and identifying potential regressions.
* Supporting performance sheriffs with tools to assist in identifying regressions.
* Developing test plans for performance testing.
* Running adhoc manual or partially automated performance testing.


= What we don't do =
= What we don't do =
* Own all performance tests. We work on the test harnesses and tools that are used for performance testing, but the tests themselves are often developed outside of our team. Every test should have an owner, who is responsible for responding to questions related to the test, and may be asked to assist when the test detects a regression.
* Maintenance of infrastructure hardware.
* Review all performance tests. Similar to test ownership, we enable others to contribute performance tests. We can provide advice and reviews, but do not impose this as a restriction to landing test changes.
* Maintain the continuous integration pipeline.
* Maintain the infrastructure the tests run on
* Writing/maintaining all performance tests.
* Maintain the continuous integration pipeline
* Maintain the reporting tools


= Meetings =
= Meetings =
Line 36: Line 36:
Welcome to the team! You are encouraged to improve the [[/Onboarding|onboarding page]]. If you need to ask questions that are not already covered, please update the page so that the next person has a better onboarding experience.
Welcome to the team! You are encouraged to improve the [[/Onboarding|onboarding page]]. If you need to ask questions that are not already covered, please update the page so that the next person has a better onboarding experience.


= Task Workflow =
= Workflow =
Visit our [[/Task Workflow|task workflow]] page for details on how we manage tasks.
* [[/Triage Process/]]
 
* [[/Review Process/]]
= Objectives =
 
== [https://docs.google.com/spreadsheets/d/1P1d7rzA27t3OHljraxapNgDl-1Hsc_novoj78w8qog0/edit#gid=584732224 2019/Q3] ==
* KR1.3: Fenix Startup Test Development [rwood] {{notstarted|}} [%]
* KR3.1: Video QoE testing for desktop [marauder] {{ok|}} [90%]
* KR3.2: Video QoE testing for Fenix [marauder] {{ok|}} [90%]
* KR3.4: Complete MVP for browsertime in CI [rwood] {{ok|}}  [100%]
* KR4.1: Migrate CI from Chromium to Chrome [sparky] {{notstarted|}} [80%]
* KR4.2: Power tests on macOS laptops [sparky] {{ok|}} [100%]
* KR6.6: Improve perf sheriff efficiency [igoldan] {{ok|}} [70%]


= Projects =
= Projects =
* [[/Fenix/]]
* [[/Raptor/]]
* [[/Raptor/Browsertime/]]
* [[/Talos/]]


== Raptor ==
= Results =
Raptor is a Python testing framework used to run browser benchmark and browser page-load performance tests. Raptor is cross-browser compatible and is currently running in Mozilla taskcluster production on Firefox Desktop, Firefox Android, and on Google Chromium.
See our {{wip|[[/Results|results page]]}}.
 
* Owner: Rob Wood [:rwood]
* Source: https://searchfox.org/mozilla-central/source/testing/raptor
* Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DRaptor
* Documentation: https://wiki.mozilla.org/Performance_sheriffing/Raptor
 
== Talos ==
Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.
 
* Owner: Rob Wood [:rwood], Joel Maher [:jmaher]
* Source: https://searchfox.org/mozilla-central/source/testing/talos
* Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DTalos
* Documentation: https://wiki.mozilla.org/Performance_sheriffing/Talos
 
== WebPageTest ==
* Owner: Stephen Donner [:stephend]
* Source: https://github.com/mozilla/wpt-api
* Good first bugs: https://github.com/mozilla/wpt-api/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22
* Documentation: https://mozilla-wpt-api-docs.readthedocs.io/en/master/
 
== Perfherder ==
Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).
Perfherder is part of the Treeherder project.
 
* Location: https://treeherder.mozilla.org/perf.html
* Owner: Ionuț Goldan [:igoldan]
* Source: https://github.com/mozilla/treeherder
* Good first bugs: https://codetribute.mozilla.org/projects/reporting?project%3DPerfherder
* User documentation: https://wiki.mozilla.org/EngineeringProductivity/Projects/Perfherder
* Developer documentation: https://treeherder.readthedocs.io/
 
= Dashboards =
 
== Firefox Are We Fast Yet Dashboard ==
Shows a variety of benchmarks, run on a variety of platforms.  Meant to be a detailed view of performance.
 
* Location: https://arewefastyet.com
* Owner: Armen Zambrano Gasparnian [:armen]
* Source: https://github.com/mozilla-frontend-infra/firefox-performance-dashboard
 
== Firefox Health Dashboard ==
The health dashboard tracks metrics and statistics important for tracking performance improvements.  Meant to be a high level view.
 
* Location: https://health.graphics/
* Owner: Armen Zambrano Gasparnian [:armen], Kyle Lahnakoski [:ekyle]
* Source: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/
* Good first bugs: https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22


= Resources =
= Resources =
Line 107: Line 54:
* [https://docs.google.com/document/d/1HV2_z8hwhI2w8EbURtkYjpikVG5g9QeKEPo9h5msuRs/edit Following up perf bugs]
* [https://docs.google.com/document/d/1HV2_z8hwhI2w8EbURtkYjpikVG5g9QeKEPo9h5msuRs/edit Following up perf bugs]
* [https://docs.google.com/document/d/103SRVVcE2SZNYP3kFXGeiVQrusH2Wj2yv8SWaDvB9SM/edit Excessive Android device queue response plan]
* [https://docs.google.com/document/d/103SRVVcE2SZNYP3kFXGeiVQrusH2Wj2yv8SWaDvB9SM/edit Excessive Android device queue response plan]
* [https://docs.google.com/document/d/1jVSaTlMZx-9DCjJurDpZJElm3yxN7N3OyjVh7hmwBb4/edit Perf bisection workflows]
* [[/FAQ#How_can_I_do_a_bisection.3F|Bisection Workflow]]
* [[/Sheriffing/CompareView|CompareView]]

Latest revision as of 17:57, 24 July 2023

Redirect to:

Fxperftest.png

New contributors

If you are a new contributor, or would like to start contributing you can find a guide to help you here.

Where to find us

Team purpose

To support the infrastructure and creation of automated tests for evaluating the performance of Firefox products. This provides value by exposing gaps in coverage, revealing areas where we can make performance gains, identifying performance regressions in a timely manner, and by providing key performance metrics that assist in determining how Firefox measures against release criteria.

What we do

  • Identification of gaps in performance test infrastructure and monitoring.
  • Designing and building performance test infrastructure and monitoring solutions.
  • Supporting Firefox engineers on writing performance tests.
  • Supporting Firefox engineers on investigating regressions identified by tests.
  • Collaboration with release operations on infrastructure requirements.
  • Standing up performance tests in continuous integration environments.
  • Monitoring performance test results and identifying potential regressions.
  • Supporting performance sheriffs with tools to assist in identifying regressions.
  • Developing test plans for performance testing.
  • Running adhoc manual or partially automated performance testing.

What we don't do

  • Maintenance of infrastructure hardware.
  • Maintain the continuous integration pipeline.
  • Writing/maintaining all performance tests.

Meetings

Onboarding

Welcome to the team! You are encouraged to improve the onboarding page. If you need to ask questions that are not already covered, please update the page so that the next person has a better onboarding experience.

Workflow

Projects

Results

See our 🚧 results page 🚧.

Resources

Retrieved from "https://wiki.allizom.org/index.php?title=TestEngineering/Performance&oldid=1247190"