Firefox Performance Test Engineering 🔥🦊⏱

From MozillaWiki
Jump to navigation Jump to search


Who we are

  • Dave Hunt [:davehunt] 🇬🇧
  • Stephen Donner [:stephend] 🇺🇸
  • Rob Wood [:rwood] 🇨🇦
  • Greg Mierzwinski [:sparky] 🇨🇦
  • Ionuț Goldan [:igoldan] 🇷🇴
  • Florian Strugariu [:bebe] 🇷🇴
  • Marian Raiciof [:marauder] 🇷🇴
  • Alex Ionescu [:alexandrui] 🇷🇴
  • Arnold Iakab [:arno__] 🇷🇴
  • Alex Irimovici [:air] 🇷🇴

Where to find us

  • IRC: #perftest
  • Slack: #perftest

What we do

We provide automated testing support for measuring the performance of Firefox products. Here are a few examples of what you can expect our team to be working on.

  • Provide advice/troubleshooting related to performance testing or any of our tools and harnesses.
  • Develop test plans that involve automation of performance testing.
  • Prototype, build, and maintain test harnesses for performance testing.
  • Monitoring and reporting of performance regressions.

What we don't do

  • Own all performance tests. We work on the test harnesses and tools that are used for performance testing, but the tests themselves are often developed outside of our team. Every test should have an owner, who is responsible for responding to questions related to the test, and may be asked to assist when the test detects a regression.
  • Review all performance tests. Similar to test ownership, we enable others to contribute performance tests. We can provide advice and reviews, but do not impose this as a restriction to landing test changes.
  • Maintain the infrastructure the tests run on
  • Maintain the continuous integration pipeline
  • Maintain the reporting tools

Onboarding

You are encouraged to improve the onboarding documentation. If you need to ask questions that are not already covered, please update the documentation so that the next person has a better onboarding experience.

Meetings

Calendar

There's a Firefox Test Engineering shared calendar (iCal), which is primarily used for PTO and related events.

PTO

Add any PTO to the shared calendar (see above) so the team are aware.

Groups

There's a perftest group, which is invite only, and used for team communications and setting up test accounts. There's also a performance group for general discussion and announcements, and a perfteam group for the performance engineering team. Feel free to sign up to these, and post to them when you have something to share or questions to ask. There is also a perf-sheriffs group for performance sheriffs. For regression summaries, you can subscribe to the pi-monitoring group.

Credentials

There's a shared 1Password vault for credentials that you may need to access. Please submit a request for 1Password from The Hub. Once you have an account and the software set up (available on iOS, Android, Windows, macOS) you can be added you to the team vault.

Additional equipment

You may need additional hardware such as mobile devices, laptops, etc. You can request this equipment from The Hub.

GitHub

We have a GitHub team for simplifying access to repositories. All team members that belong to the Mozilla organisation should be added to this team as maintainers. Other team members will need to be manually granted access to individual repositories as needed.

Shared folder

We have a shared folder in Google Drive.

Sheriffing

Performance sheriffs will need to complete the following:

Review policy

When you push a commit up for review, you should use the following syntax to request review from the perftest review group:

r=#perftest

A single r+ from one reviewer is required for the patch to be allowed to be sent off for integration. More reviewers can pitch in on the same review, and Lando will in this case automatically rewrite the commit message to show who was involved signing off the patch, for example:

Bug 1546611 - Fix "None" checks when validating test manifests; r=perftest,rwood

When you occasionally you have to single out individuals for specific topic expertise, you add an exclamation mark behind the nickname:

r=#perftest,rwood!

This will add the patch to the shared review queue, but also block the review from landing pending Rob's approval. Requested changes by other reviewers will also block the review.

Task Workflow

New requests

New requests can be created via Jira Service Desk (requires account), or Mana.

Triage

Occurs weekly on Mondays at 14:45 UTC using the Untriaged filter.

  1. Set status to Current Quarter if work is to be considered for the current quarter
    • Set due date if there is a deadline, else use last working day in quarter
    • Set priority if known to be high or low, else use medium
    • Set assignee to be responsible for planning
    • Set status to Future Quarter if work is to be considered for next quarter planning
  2. Set deferred date to bump to a future triage date

Current quarter planning

Occurs regularly using the Needs Planning (me) and Needs Planning filters.

  1. Create links for blockers, dependencies, etc
  2. Create subtasks if needed, for each subtask:
    • Set priority if known to be high or low, else use medium
    • Set assignee to be responsible for development
    • Set due date if there is a deadline, else reflect parent task due date
    • Create links for blockers, dependencies, etc
    • Set time estimate (in weeks)
    • If no further planning is needed, set status to Ready
  3. If no subtasks are created:
    • Set assignee to be responsible for development
    • Set time estimate (in weeks)
  4. If no further planning is needed, set status to Ready

Weekly review

Occurs weekly on Mondays at 15:30 UTC using the Blocked, Overdue and Upcoming filters.

  1. Review due date, priority, and status

Estimates

Occurs regularly using the Needs Estimate (me) and Needs Estimate filters.

  1. Set time estimate (in weeks)

Next quarter planning

Occurs each quarter using the Future Quarter filter.

  1. To be considered for the upcoming quarter:
    • Set status to Current Quarter
    • Set due date if there is a deadline, else use last working day in next quarter
    • Set priority if known to be high or low, else use medium
    • Set assignee to be responsible for planning
  2. To be reconsidered for a future quarter leave status as Future Quarter

Current quarter review

Occurs regularly using the Current Quarter filter.

  1. Determine time (weeks) remaining in quarter
  2. Review assignee, due date, priority, and status

Development

Occurs whenever we are free to take on new work, using the Ready (me) and Ready filters.

  1. Set status to Dev
  2. Raise a tracking bug or issue against the appropriate project
  3. Add link to the tracking bug or issue

Blocked

Occurs whenever a task is blocked.

  1. Set status to Blocked
  2. If task is blocked by other task(s):
    • Add link(s) to the blocking task(s)
  3. Add a comment detailing the circumstances of the blocker(s).

Objectives

2019/H1

  • Increase regression detection coverage to page load on Android products including WebView comparisons
  • Increase quality standards by measuring and alerting on power usage for Android and ARM64
  • Build dashboards to improve visibility of how we are tracking against our release criteria for Android
  • Reducing noise in test results and adding annotations to provide clarity of signal for regressions
  • Develop measurements and mechanisms for reporting on our tools, policies and documentation for how to improve clarity and efficiency of risk assessments

2019/Q2

  • Meet Fenix blocking performance release criteria
    • Fenix 1.0 geo mean cold page load time to onload event is >20% faster than Fennec 64 for tp6m reference sites (2019Q1 snapshots) on Fenix reference hardware
  • Perform well on priority use cases for Qualcomm's H2 product launches
    • Firefox v68 for Windows 10 on ARM battery utilization is within 20% of Edge on Qualcomm 8xx series reference hardware when watching the tp6m YouTube reference video to completion
    • The proportion of dropped frames during playback of the tp6m YouTube reference video at 1080p resolution does not exceed 4% on Firefox v68 on the Qualcomm 8xx series reference hardware.
    • Firefox v68 for Windows 10 on ARM geomean cold page load time to the onload event is within 20% of Edge for tp6 reference sites on Qualcomm 8xx series reference hardware.
  • Meet blocking performance objectives for GeckoView-powered Fire TV pitch
    • The proportion of dropped frames during playback of the tp6m 4K YouTube reference video using GeckoView-powered Firefox for Fire TV does not exceed 4% on the Fire TV 4k stick.
  • Lay foundation to support performance work in H2
    • Develop key metrics and scenarios needed to build CI-integrated tests for at least three of video quality, scroll responsiveness, input responsiveness, memory use, CPU utilization, and heat.
  • Make the Firefox desktop browser feel fast and responsive

Projects

Raptor

Raptor is a Python testing framework used to run browser benchmark and browser page-load performance tests. Raptor is cross-browser compatible and is currently running in Mozilla taskcluster production on Firefox Desktop, Firefox Android, and on Google Chromium.

Talos

Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.

WebPageTest

Perfherder

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android). Perfherder is part of the Treeherder project.

Dashboards

Firefox Are We Fast Yet Dashboard

Shows a variety of benchmarks, run on a variety of platforms. Meant to be a detailed view of performance.

Firefox Health Dashboard

The health dashboard tracks metrics and statistics important for tracking performance improvements. Meant to be a high level view.

Resources

Retrieved from "https://wiki.allizom.org/index.php?title=TestEngineering/Performance&oldid=1213790"