Electrolysis/Release Criteria

From MozillaWiki
< Electrolysis
Revision as of 17:38, 3 March 2016 by Benjamin Smedberg (talk | contribs) (add milestone to blocking query)
Jump to navigation Jump to search

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

Release Criteria

User Engagement

Responsible glind? dzeber? hulmer?
Accountable TBD
Supporting measurement/data teams
Consulted jgriffiths, release management
Informed cpeterson, elan
Release Criteria Metric Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
User Retention Experiment user drop-off rate (?) bug 1249665 glind? TBD TBD NO ONE
User Retention e10s/non-10s engagement ratio bug 1251259 dzeber, hulmer? TBD TBD NO ONE

Stability

Responsible chutten
Accountable bsmedberg
Supporting Kairo, measurement/data teams
Consulted jgriffiths, release management
Informed cpeterson, elan
Release Criteria Metric Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
chrome+content process crash rate chrome process crash ping counts + SUBPROCESS_CRASHES_WITH_DUMP['content'] per 1000 hours of subsessionLength IPython notebook rvitillo TBD NO NO ONE
plugin process crash rate SUBPROCESS_CRASHES_WITH_DUMP['plugin'] per 1000 hours of subsessionLength IPython notebook rvitillo TBD NO NO ONE

Jank

Responsible chutten
Accountable bsmedberg
Supporting data team, RyanVM, rvitillo, avih, Softvision
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
GC pauses GC_MAX_PAUSE_MS beta 45 experiment TBD TBD TBD TBD
Cycle Collector pauses CYCLE_COLLECTOR_MAX_PAUSE beta 45 experiment TBD TBD TBD TBD
Event loop lag INPUT_EVENT_RESPONSE_MS TBD TBD TBD TBD TBD
Browser responsiveness during page load Talos tp5o_responsiveness Perfherder TBD TBD About -87% faster! TBD
Frame delay FX_REFRESH_DRIVER_{CHROME,CONTENT}_FRAME_DELAY_MS beta 45 experiment TBD TBD TBD TBD
BHR/chrome hangs TBD TBD TBD TBD TBD TBD

Scrolling

Responsible kats
Accountable RyanVM
Supporting avih, jmaher, graphics team (mstange)
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Metric Metric Owner Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS kats? bug 1228147? kats? TBD TBD TBD
tp5o_scroll avih? mstange? Perfherder (bug 1179735) kats? TBD Up to 105% worse! TBD
tp5r kats? TBD kats? TBD TBD TBD
tscrollx avih? mstange? Perfherder (bug 1174772? bug 1249976?) kats? TBD Up to 32% worse! TBD

Slow Scripts

Responsible azhang
Accountable bsmedberg
Supporting
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
Slow script warnings SLOW_SCRIPT_PAGE_COUNT (bug 1251667) TBD TBD TBD TBD TBD

UI Smoothness

Responsible mconley
Accountable RyanVM
Supporting jmaher, avih, gabor
Consulted jgriffiths
Informed cpeterson, elan, release management
Metric Metric Owner Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test) mconley Perfherder TBD TBD Up to 17% worse TBD
tpaint ±5% (first paint for a new window) jimm Perfherder TBD TBD Up to 46% worse TBD
tresize ±5% (window resize responsiveness) jimm Perfherder TBD TBD Mixed results: up to 5.44% worse TBD
tps ±5% (tab switch speed) mconley Perfherder TBD TBD Up to 107% worse! TBD
FX_TAB_SWITCH_TOTAL_MS mconley? TBD TBD TBD TBD TBD
CART ±5% (Customize Animation Regression Test) mconley Perfherder TBD TBD Up to 14% worse TBD

Page Load

Responsible jimm
Accountable RyanVM
Supporting jmaher
Consulted jgriffiths, rvitillo
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
Talos tests for page load time tp5o ±5% Perfherder no change TBD TBD TBD
Page load time telemetry FX_PAGE_LOAD_MS ±5% beta 45 experiment rvitillo TBD TBD NO ONE

Plugin Jank

Responsible chutten
Accountable bsmedberg
Supporting aklotz
Consulted jgriffiths
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
Plugin initialization jank time BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5% TBD TBD TBD TBD TBD
Plugin destruction jank time BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5% TBD TBD TBD TBD TBD


Graphics Performance

Responsible Milan
Accountable RyanVM
Supporting gw280
Consulted jgriffiths, jmaher, graphics team
Informed cpeterson, elan, release management
Metric Metric Owner Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
glterrain ±5% jgilbert Perfherder TBD TBD Mixed results: from 4% worse to 63% better TBD
tcanvasmark ±5% TBD Perfherder TBD TBD From 2% worse to 3% better TBD
tsvgx ±5% mstange? Perfherder TBD TBD Up to 51% better TBD
tsvg_opacity ±5% avih says no real owner. mattwoodrow? seth? Perfherder TBD TBD Up to -79% better TBD

Startup/Shutdown Time

Responsible rvitillo?
Accountable
Supporting
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
Talos tests ts_paint ±10% Perfherder TBD TBD Up to 55% better TBD
Time to first paint SIMPLE_MEASURES_FIRSTPAINT ±10% beta 45 experiment bad probability #'s TBD TBD TBD
Session restore time SIMPLE_MEASURES_SESSIONRESTORED ±10% beta 45 experiment no data? TBD TBD TBD
Shutdown time SIMPLE_MEASURES_SHUTDOWNDURATION ±25% beta 45 experiment no data? TBD TBD TBD
  • Startup times may regress up to 10% and shutdown time up to 25% compared to non-e10s.

Memory Usage

Responsible chutten
Accountable bsmedberg
Supporting
Consulted jgriffiths
Informed cpeterson, elan, release management
Metric Metric Owner Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
tp5o Main_RSS Perfherder TBD TBD 4% to 35% better TBD
tp5o Private Bytes Perfherder TBD TBD 20% to 50% worse TBD
Release Criteria Metric Analysis Analysis By Analysis Signed-off By Criteria Met? Criteria Met Signed-off By
Swap Rate (TBD) bug 1198211 TBD chutten TBD TBD TBD
Overall memory usage (browser + content) bug 1198209 TBD chutten TBD TBD TBD

Release Blocking Bugs

All M8 and M9 bugs must be fixed before we release.

Responsible jimm?
Accountable blassey?
Supporting release management
Consulted jgriffiths
Informed cpeterson, elan

M8/M9 bugs:

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Release Criteria meta bugs' blockers:

Full Query
ID Summary Whiteboard Keywords Assigned to
1250350 Talos svg results for e10s seem invalid
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Tests

All tests disabled for e10s must be triaged and:

  • reenabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s?
  • or deleted?
Responsible Blake, Felipe
Accountable blassey?
Supporting
Consulted jgriffiths
Informed cpeterson, elan, release management


Detail Pages

Release Criteria Responsible Status Tracking Bug
P1: Stability poiru [AT RISK] bug 1251376
P1: Jank chutten TBD bug 1251377
P1: Scrolling kats [AT RISK] bug 1251388
P1: Slow Script azhang TBD bug 1251545
P1: no blocker bugs blassey [ON TRACK] m9 bug query
P1: automated tests running and passing mrbkap? [AT RISK] bug e10s-tests
P2: UI Smoothness mconley [AT RISK] bug 1251547
P2: Page Load Time jimm [ON TRACK] bug 1251549
P2: Plugin Jank chutten TBD bug 1251551
P2: Graphics Milan [ON TRACK] bug 1251555
Retrieved from "https://wiki.allizom.org/index.php?title=Electrolysis/Release_Criteria&oldid=1119880"