Firefox Performance Test Engineering 🔥🦊⏱

From MozillaWiki
Jump to navigation Jump to search


Who we are

  • Dave Hunt [:davehunt] 🇬🇧
  • Stephen Donner [:stephend] 🇺🇸
  • Rob Wood [:rwood] 🇨🇦
  • Greg Mierzwinski [:sparky] 🇨🇦
  • Ionuț Goldan [:igoldan] 🇷🇴
  • Florian Strugariu [:bebe] 🇷🇴
  • Marian Raiciof [:marauder] 🇷🇴
  • Alex Ionescu [:alexandrui] 🇷🇴
  • Alex Irimovici [:air] 🇷🇴

Where to find us

  • IRC: #perftest
  • Slack: #perftest

What we do

We provide automated testing support for measuring the performance of Firefox products. Here are a few examples of what you can expect our team to be working on.

  • Provide advice/troubleshooting related to performance testing or any of our tools and harnesses.
  • Develop test plans that involve automation of performance testing.
  • Prototype, build, and maintain test harnesses for performance testing.
  • Monitoring and reporting of performance regressions.

What we don't do

  • Own all performance tests. We work on the test harnesses and tools that are used for performance testing, but the tests themselves are often developed outside of our team. Every test should have an owner, who is responsible for responding to questions related to the test, and may be asked to assist when the test detects a regression.
  • Review all performance tests. Similar to test ownership, we enable others to contribute performance tests. We can provide advice and reviews, but do not impose this as a restriction to landing test changes.
  • Maintain the infrastructure the tests run on
  • Maintain the continuous integration pipeline
  • Maintain the reporting tools

Meetings

Onboarding

You are encouraged to improve the onboarding page. If you need to ask questions that are not already covered, please update the page so that the next person has a better onboarding experience.

Task Workflow

New requests

New requests can be created via Jira Service Desk (requires account), or Mana.

Triage

Occurs weekly on Mondays at 14:45 UTC using the Untriaged filter.

  1. Set status to Current Quarter if work is to be considered for the current quarter
    • Set due date if there is a deadline, else use last working day in quarter
    • Set priority if known to be high or low, else use medium
    • Set assignee to be responsible for planning
    • Set status to Future Quarter if work is to be considered for next quarter planning
  2. Set deferred date to bump to a future triage date

Current quarter planning

Occurs regularly using the Needs Planning (me) and Needs Planning filters.

  1. Create links for blockers, dependencies, etc
  2. Create subtasks if needed, for each subtask:
    • Set priority if known to be high or low, else use medium
    • Set assignee to be responsible for development
    • Set due date if there is a deadline, else reflect parent task due date
    • Create links for blockers, dependencies, etc
    • Set time estimate (in weeks)
    • If no further planning is needed, set status to Ready
  3. If no subtasks are created:
    • Set assignee to be responsible for development
    • Set time estimate (in weeks)
  4. If no further planning is needed, set status to Ready

Weekly review

Occurs weekly on Mondays at 15:30 UTC using the Blocked, Overdue and Upcoming filters.

  1. Review due date, priority, and status

Estimates

Occurs regularly using the Needs Estimate (me) and Needs Estimate filters.

  1. Set time estimate (in weeks)

Next quarter planning

Occurs each quarter using the Future Quarter filter.

  1. To be considered for the upcoming quarter:
    • Set status to Current Quarter
    • Set due date if there is a deadline, else use last working day in next quarter
    • Set priority if known to be high or low, else use medium
    • Set assignee to be responsible for planning
  2. To be reconsidered for a future quarter leave status as Future Quarter

Current quarter review

Occurs regularly using the Current Quarter filter.

  1. Determine time (weeks) remaining in quarter
  2. Review assignee, due date, priority, and status

Development

Occurs whenever we are free to take on new work, using the Ready (me) and Ready filters.

  1. Set status to Dev
  2. Raise a tracking bug or issue against the appropriate project
  3. Add link to the tracking bug or issue

Blocked

Occurs whenever a task is blocked.

  1. Set status to Blocked
  2. If task is blocked by other task(s):
    • Add link(s) to the blocking task(s)
  3. Add a comment detailing the circumstances of the blocker(s).

Objectives

2019/Q3

  • KR1.3: Fenix Startup Test Development [rwood] [NOT STARTED] [%]
  • KR3.1: Video QoE testing for desktop [marauder] [ON TRACK] [90%]
  • KR3.2: Video QoE testing for Fenix [marauder] [ON TRACK] [90%]
  • KR3.4: Complete MVP for browsertime in CI [rwood] [ON TRACK] [100%]
  • KR4.1: Migrate CI from Chromium to Chrome [sparky] [NOT STARTED] [80%]
  • KR4.2: Power tests on macOS laptops [sparky] [ON TRACK] [100%]
  • KR6.6: Improve perf sheriff efficiency [igoldan] [ON TRACK] [70%]

2019/H1

  • Increase regression detection coverage to page load on Android products including WebView comparisons
  • Increase quality standards by measuring and alerting on power usage for Android and ARM64
  • Build dashboards to improve visibility of how we are tracking against our release criteria for Android
  • Reducing noise in test results and adding annotations to provide clarity of signal for regressions
  • Develop measurements and mechanisms for reporting on our tools, policies and documentation for how to improve clarity and efficiency of risk assessments

2019/Q2

  • Meet Fenix blocking performance release criteria
    • Fenix 1.0 geo mean cold page load time to onload event is >20% faster than Fennec 64 for tp6m reference sites (2019Q1 snapshots) on Fenix reference hardware
  • Perform well on priority use cases for Qualcomm's H2 product launches
    • Firefox v68 for Windows 10 on ARM battery utilization is within 20% of Edge on Qualcomm 8xx series reference hardware when watching the tp6m YouTube reference video to completion
    • The proportion of dropped frames during playback of the tp6m YouTube reference video at 1080p resolution does not exceed 4% on Firefox v68 on the Qualcomm 8xx series reference hardware.
    • Firefox v68 for Windows 10 on ARM geomean cold page load time to the onload event is within 20% of Edge for tp6 reference sites on Qualcomm 8xx series reference hardware.
  • Meet blocking performance objectives for GeckoView-powered Fire TV pitch
    • The proportion of dropped frames during playback of the tp6m 4K YouTube reference video using GeckoView-powered Firefox for Fire TV does not exceed 4% on the Fire TV 4k stick.
  • Lay foundation to support performance work in H2
    • Develop key metrics and scenarios needed to build CI-integrated tests for at least three of video quality, scroll responsiveness, input responsiveness, memory use, CPU utilization, and heat.
  • Make the Firefox desktop browser feel fast and responsive

Projects

Raptor

Raptor is a Python testing framework used to run browser benchmark and browser page-load performance tests. Raptor is cross-browser compatible and is currently running in Mozilla taskcluster production on Firefox Desktop, Firefox Android, and on Google Chromium.

Talos

Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.

WebPageTest

Perfherder

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android). Perfherder is part of the Treeherder project.

Dashboards

Firefox Are We Fast Yet Dashboard

Shows a variety of benchmarks, run on a variety of platforms. Meant to be a detailed view of performance.

Firefox Health Dashboard

The health dashboard tracks metrics and statistics important for tracking performance improvements. Meant to be a high level view.

Resources

Retrieved from "https://wiki.allizom.org/index.php?title=TestEngineering/Performance&oldid=1215934"