MozillaWiki

Electrolysis/Release Criteria

< Electrolysis
Revision as of 16:39, 27 May 2016 by Cpeterson (talk | contribs) (→‎Stability: Link to stability dashboard)

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

Release Criteria

Stability

Responsible bsmedberg
Accountable bsmedberg
Supporting Kairo, rvitillo
Consulted Barbara, release management
Informed cpeterson, elan
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
chrome+content process crash rate chrome process crash ping counts + SUBPROCESS_CRASHES_WITH_DUMP['content'] per 1000 hours of subsessionLength 46 experiment rvitillo 84,390 (Beta 45) 109,348 (Beta 45) NO TODO
plugin process crash rate SUBPROCESS_CRASHES_WITH_DUMP['plugin'] per 1000 hours of subsessionLength 46 experiment rvitillo 41,897 (Beta 45) 55,809 (Beta 45) NO TODO

User Engagement

Responsible rvitillo
Accountable TBD
Supporting measurement/data teams
Consulted Barbara, release management
Informed cpeterson, elan
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Engagement Ratio (DAU/MAU) Engagement Ratio graphs (bug 1251259) rvitillo ~0.22 ~0.35 Yes rvitillo, bsmedberg (2016-05-27)

Jank

Responsible chutten
Accountable bsmedberg
Supporting data team, RyanVM, rvitillo, avih, Softvision
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
GC pauses GC_MAX_PAUSE_MS (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Cycle Collector pauses CYCLE_COLLECTOR_MAX_PAUSE (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Event loop lag INPUT_EVENT_RESPONSE_MS beta 46 experiment

Nightly 48 telemetry

chutten e10s is different but not worse chutten and jgriffiths (2016-04-08)
Browser responsiveness during page load Talos tp5o_responsiveness Perfherder YES: e10s is 39%–88% faster chutten and jgriffiths (2016-04-08)

Page Load

Responsible jimm
Accountable RyanVM
Supporting jmaher
Consulted jgriffiths, rvitillo
Informed cpeterson, elan, release management
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o summary ±5% Perfherder 248.32 (Linux)

230.35 (Win8)

246.28 (Linux)

213.51 (Win8)

YES: -0.82% to -7.31% improvement jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

FX_PAGE_LOAD_MS ±5% beta 46 experiment (bug 1228020) rvitillo e10s is much better. YES: e10s is much better. jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

Startup/Shutdown Time

Responsible rvitillo
Accountable
Supporting
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Talos tests ts_paint ±10% Perfherder 863 (Win7) 433 (Win7) YES: 43–55% better (bug 1174767) jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Time to first paint SIMPLE_MEASURES_FIRSTPAINT ±10%

("simpleMeasurements/firstPaint")

beta 46 experiment rvitillo 4468 ms 4391 ms YES: e10s is slightly better jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Shutdown time SIMPLE_MEASURES_SHUTDOWNDURATION ±25%

("simpleMeasurements/shutdownDuration")

beta 46 experiment rvitillo 1156 ms 1190 ms YES: e10s is slightly worse, but within ±25% tolerance. jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

  • In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".

Scrolling

Responsible kats
Accountable RyanVM
Supporting avih, jmaher, graphics team (mstange)
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS beta 46 experiment avih YES: e10s is faster kats and jgriffiths (2016-04-08)
tp5o_scroll Perfherder kats YES: e10s is 6% – 53% better kats and jgriffiths (2016-04-08)
tscrollx Perfherder kats YES: e10s is 7% – 58% better kats and jgriffiths (2016-04-08)

Plugin Jank

Responsible chutten
Accountable aklotz
Supporting bsmedberg
Consulted Barbara
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
Plugin module initialization jank time BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is better.

Java: YES. e10s is better.

Silverlight: YES. e10s is better.

Unity: YES. e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin initialization jank time BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5% Beta 46 experiment chutten Flash: MAYBE? e10s is slightly worse.

Java: YES. e10s is slightly better.

Silverlight: MAYBE? e10s is slightly worse.

Unity: YES! e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin destruction jank time BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is different, but not worse.

Java: MAYBE? e10s is slightly worse.

Silverlight: MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)

Unity: YES. e10s is different, but not worse.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Memory Usage

Responsible erahm
Accountable blassey
Supporting jmaher (Talos)
Consulted Barbara
Informed cpeterson, elan, release management
Metric Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o Main_RSS Perfherder 401623141.33 (OS X) 331418381.64 (OS X) e10s is 17%–35% better erahm (2016-04-15), Barbara (2016-04-18)
MEMORY_TOTAL ±25% (browser + content process) beta 46 experiment and erahm's blog post e10s is 10–20% worse, but within our ±25% tolerance erahm (2016-04-15), Barbara (2016-04-18)
VSIZE_MAX_CONTIGUOUS (browser + content process) beta 46 experiment e10s is slightly different, but not worse? erahm (2016-04-15), Barbara (2016-04-18)

UI Smoothness

Responsible mconley
Accountable RyanVM
Supporting jmaher, avih, gabor
Consulted Barbara
Informed cpeterson, elan, release management
Metric Metric Owner Analysis Measurements Bugs Criteria Met? Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test) mconley Perfherder

e10s / APZ
xp = 4.4% combined
win7 = 1.8% / 9%
win8 = -0.6% / 10% (improvement)
osx = 6% / 3.5%
linux = 0.5% / 16%

e10s: bug 1251699
apz:bug 1253412

Borderline mconley (2016-04-19), Barbara (2016-04-20), avih (2016-04-21)
tpaint ±5% (first paint for a new window) mconley Perfherder

xp = 0%
win7 = 9.3%
win8 = 3.4%
osx = -11% (improvement)
linux = 7%

bug 1174770 Borderline mconley (2016-04-19), avih (2016-04-21), Barbara (2016-04-22)
tresize ±5% (window resize responsiveness) jimm Perfherder

e10s / APZ
xp = 1.0% combined
win7 = 0.0%
win8 = -22% (improvement overall)
osx = 1.0% combined
linux = 7% / 2.0%

bug 1252947 Yes jgriffiths (2016-03-17, bug 1252947#c23), mconley (2016-04-21)
tps ±5% (tab switch speed) mconley Perfherder

xp = -27%
win7 = -28%
win8 = -34%
osx = -30%
linux = -19%

e10s: bug 1186585
apz: bug 1216924

Yes mconley (2016-04-19), Barbara (2016-04-20)
FX_TAB_SWITCH_TOTAL_MS Gabor beta 46 experiment 52.66 ms (median) < 82.61 ms (median) bug 1252031 Yes gabor (2016-03-23, bug 1252031#c2), Barbara (2016-04-21)
CART ±5% (Customize Animation Regression Test) mconley Perfherder

e10s / APZ
xp = improvement
win7 = 4.0% / 2.5%
win8 = 2.0% / 2.5%
osx = 3.7% / 2.7%
linux = 6.8% / 6.8%

e10s: bug 1252768
apz: bug 1253380

Borderline avih (2016-04-21), mconley (2016-04-19), Barbara (2016-04-22)

Graphics Performance

Responsible Milan
Accountable RyanVM
Supporting gw280
Consulted Barbara, jmaher, graphics team
Informed cpeterson, elan, release management
Metric Metric Owner Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
glterrain ±5% jgilbert Perfherder 18 19 Mixed results from 3% worse to -66% better Barbara (2016-05-06), Milan (2016-05-06)
tcanvasmark ±5% Perfherder 6444 6301 Mixed results from -2% worse to 3% better Barbara (2016-05-06), Milan (2016-05-06)

Slow Scripts

Responsible azhang
Accountable bsmedberg
Supporting
Consulted Barbara, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Slow script warnings SLOW_SCRIPT_PAGE_COUNT beta 47 experiment azhang 0.00117087669174 warnings per hour 0.00158655016424 warnings per hour

(bug 1260769)

e10s is good enough. Barbara and jimm are OK with this 35% regression in e10s. Barbara, jimm, billm (2016-05-25)

Tests

All tests disabled for e10s must be triaged and:

  • reenabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s?
  • or deleted?

We want each test directory to be signed-off by the feature area's owner.

Responsible Blake, Felipe
Accountable jimm
Supporting dolske (coordinating people fixing front-end tests)
Consulted jgriffiths
Informed cpeterson, elan, release management
Task Owner Criteria Met? Criteria Met Signed-off By
Write test plan RyanVM In progress: RyanVM says he is waiting for feedback from blassey, jimm, and milan. TODO
Execute test plan RyanVM In progress TODO
Re-enable disabled e10s tests Felipe, Blake In progress TODO

Release Blocking Bugs

All M9 bugs must be fixed before we release.

Responsible jimm
Accountable blassey
Supporting release management
Consulted jgriffiths
Informed cpeterson, elan

M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Release Criteria bugs

Full Query
ID Summary Whiteboard Keywords Assigned to
1063169 [meta] tracking performance issues with e10s [webvr] meta
1250350 Talos svg results for e10s seem invalid
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);


APZ Bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Accessibility

  • Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.

Add-ons

  • Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.
Retrieved from "https://wiki.allizom.org/index.php?title=Electrolysis/Release_Criteria&oldid=1134750"