Electrolysis/Release Criteria/Jank

Bug 1251377 - [e10s release criteria] Jank, responsiveness should not regress

RASCI:

• Responsible: chutten
• Accountable: bsmedberg
• Supporting: data team, RyanVM, rvitillo, avih, Softvision
• Consulted:
• Informed: cpeterson, elan, release management

Technical proposal:

The following metrics will be compared with e10s and non-e10s A/B tests:

FX_REFRESH_DRIVER_{CHROME,CONTENT}_FRAME_DELAY_MS

Note from billm: bug 1228147 seems invalid because it considers users outside the experiment.

This measures the delay from when the underlying platform informs us of a vsync edge to when we handle it on the main thread of the stated process. As such, this is a reasonable measure of how main thread lag influences perceived jank (since it measures some of how long it takes changed pixels to show on-screen).

• In non-e10s, CHROME is the only measure with values.
• In e10s, CHROME and CONTENT both have values.

These metrics are only useful if the distribution of paint requests prompting the measures remains comparable. Both e10s and APZ change the distribution (the first by splitting the work and changing what work is performed in what process, the second by changing how many scrolling events are measured by these metrics), meaning that these measures are not able to be meaningfully compared between cohorts that have different e10s or APZ settings.

BHR/Chrome hangs

We have concerns about the accuracy of the data being collected for each of these measures, see bug 1240887. But we have agreed to accept the existing analysis which says that BHR and chromehangs improved with e10s and consider this requirement PASSed.

Followup may be required if BHR data is used to validate future addon-related jank.

Please provide: links to the bugs/spark analyses.

Event loop lag

Originally proposed: EVENTLOOP_UI_ACTIVITY_EXP_MS

INPUT_EVENT_RESPONSE_MS is the better measure for e10s/nonE10s comparisons, as it is valid across more than one OS and more than one process (EVENTLOOP_UI_ACTIVITY_EXP_MS is valid on Windows only and in the chrome process only). I was using the analysis of that measure as the primary reason for closing bug 1223780 ( https://bugzilla.mozilla.org/show_bug.cgi?id=1223780 ) using analyses on beta45ex1 (prelim analysis was done here: https://gist.github.com/chutten/9b9e29df10e0f7306f99 analysis on the later data was performed, but not published, as it was largely identical) and prelim data from beta45ex2 ( https://gist.github.com/chutten/3129baf8d5e0f10ef54a )

Open questions:

• has the new metric recieved appropriate testing? (automated or manual)
• Do we need to follow up and do camera-based analysis to measure latency from keypress to actual visual on-screen?
• are there any concerns? Can we call this a PASS?

jank per minute of active usage

This is a combined metric, bug 1198650. We have made the decision that this no longer blocks e10s, because we are looking at the individual components.

Talos tp5o_responsiveness

• e10s comparison validated: jimm
• Current e10s diff: much better - ~90% on all platforms
  • No results on OS X
• Note: measures browser responsiveness during page load. In e10s measured only at the chrome process, therefore the improvement seems real. It would still be useful to also collect data for the content process. TBD.
• bug 631571 - add the test to talos
• bug 710296 - enable the test in e10s (later comments)

GC pauses

GC_MAX_PAUSE_MS, CYCLE_COLLECTOR_MAX_PAUSE e10s does better here PASS ?