Electrolysis/Release Criteria/Slow Script: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Add bug list)
(Update with latest info from experiments.)
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
e10s must not cause the slow-script dialog to appear more often
e10s must not cause the slow-script dialog to appear more often. A/B testing on beta should indicate that the slow script dialog appears no more often than before.


* a/b testing on beta should indicate that the slow script dialog appears no more often than before
= RASCI =
 
RASCI:
* Responsible: azhang
* Responsible: azhang
* Accountable: bsmedberg
* Accountable: bsmedberg
* Supporting:  
* Supporting:  
* Consulted:  
* Consulted:  
* Informed: cpeterson, elan
* Informed: cpeterson, elan, release management
 
= Metrics =
 
== SLOW_SCRIPT_PAGE_COUNT ==
 
Summary:
* SLOW_SCRIPT_PAGE_COUNT made it into beta46, and we've obtained data for it in the beta46-noapz and beta46-apz experiment.
* See the [https://github.com/vitillo/e10s_analyses/blob/master/beta46-apz/e10s_experiment.ipynb latest beta46-apz experiment analysis]: SLOW_SCRIPT_PAGE_COUNT still shows slightly more often on e10s than non-e10s, but the difference is much smaller than with SLOW_SCRIPT_NOTICE_COUNT.
* We've verified that SLOW_SCRIPT_PAGE_COUNT measures the same thing across e10s and non-e10s.


Summary
Relevant info from beta46-apz slow script analysis:
* It seems that there is indeed a higher number of slow-script notices per session on e10s.
* We've verified that we're measuring the same thing across e10s and non-e10s.
* Does this imply that e10s has more slow-script notifications? Possible confounding factors:
** The dialog on e10s can potentially stop and re-show multiple times (theorized by :jimm). For example, if facebook.com freezes the content process for 12 seconds every minute, non-e10s will block while showing the dialog, while e10s will show the dialog, hide it after 12 seconds, and then reshow it, every minute.
** Slow script notices can occur in multiple tabs at once in e10s. This seems like a rare occurrence, however, so it shouldn't be significant.
* Code like '''setInterval(function(){var x=Date.now();while(Date.now()-x<12000){}},30000)''' can show the slow script dialog on e10s every 30 seconds, while on non-e10s it will only show it once until you select an option in the dialog. It's not really possible to account for what the script might have done on non-e10s while the slow script dialog is shown and the user hasn't selected an option yet.
* :azhang thinks the measure isn't comparable this way - in the non-e10s case, the event triggers under slightly different conditions.


Next steps:
'''Median difference in payload/histograms/SLOW_SCRIPT_PAGE_COUNT per hour is 0.12, (0.47, 0.35).
* See if it's possible to make the measure comparable between e10s/non-e10s, or mark it as a non-blocker if not possible.
The probability of this effect being purely by chance is 0.00.'''


Technical details:
Technical details:
* SLOW_SCRIPT_NOTICE_COUNT is the telemetry histogram
* SLOW_SCRIPT_PAGE_COUNT is essentially a version of SLOW_SCRIPT_NOTICE_COUNT that can be compared between e10s and non-e10s. It counts only the first slow script notice per page, which accounts for the difference in notice behaviours.
* There should be no differences in the slow script '''reporting''' mechanism between e10s and non-e10s; 1 call to ShowSlowScriptDialog means 1 addition to the histogram regardless of e10s setting. Manual testing supports this.
* There should be no differences in the slow script '''reporting''' mechanism between e10s and non-e10s; 1 call to ShowSlowScriptDialog means 1 addition to the histogram regardless of e10s setting. Manual testing supports this.
* The timeout has changed from 2 to 10 in e10s builds (see https://hg.mozilla.org/mozilla-central/rev/63680efe6d55#l35.12 for before and https://hg.mozilla.org/mozilla-central/rev/5002e9c06365#l2.13 for after). However, the value is correct on all beta versions, from 45.0b1 to 45.0b8 - the timeouts are identical between e10s and non-e10s.
* The timeout has changed from 2 to 10 in e10s builds (see https://hg.mozilla.org/mozilla-central/rev/63680efe6d55#l35.12 for before and https://hg.mozilla.org/mozilla-central/rev/5002e9c06365#l2.13 for after). However, the value is correct on all beta versions, from 45.0b1 to 45.0b8 - this does not affect our measurements.
* :azhang has verified that beta 45 releases with/without e10s both seem to increment SLOW_SCRIPT_NOTICE_COUNT if and only if there is a slow script notification shown, and that they seem to have the exact same timeout in all cases.
* :azhang has verified that beta 46 releases with/without e10s both increment SLOW_SCRIPT_PAGE_COUNT and SLOW_SCRIPT_NOTICE_COUNT if and only if there is a slow script notification shown, and that they seem to have the exact same timeout in all cases.


Links:
Links:
* {{bug|1251667}} - [e10s] Add new telemetry probe to count pages with one or more slow script notes (SLOW_SCRIPT_PAGE_COUNT?)
* https://github.com/vitillo/e10s_analyses/blob/master/beta46-apz/e10s_experiment.ipynb - e10s-beta46-apz experiment main analysis, (this has a section for slow scripts).
* https://github.com/vitillo/e10s_analyses/blob/master/beta46-noapz/e10s_experiment.ipynb - e10s-beta46-noapz experiment main analysis, (this has a section for slow scripts).
* https://gist.github.com/Uberi/64be9b7946849fa45e50 - slow script analysis
* {{bug|1249845}} - tracking bug for the beta46-apz experiment
* {{bug|1253669}} - tracking bug for the beta46-noapz experiment
* {{bug|1249978}} - [e10s] SLOW_SCRIPT_NOTICE_COUNT regressed in beta experiment
* {{bug|1249978}} - [e10s] SLOW_SCRIPT_NOTICE_COUNT regressed in beta experiment
* https://gist.github.com/Uberi/f8591cf68aab36bbb758 - notebook prepared by azhang
* {{bug|1182638}} - [e10s][telemetry] slow-script dialog appears twice as often in e10s vs non-e10s
* {{bug|1182638}} - [e10s][telemetry] slow-script dialog appears twice as often in e10s vs non-e10s


Analysis results: SLOW_SCRIPT_NOTICE_COUNT shows up significantly more often in e10s.
Analysis results: SLOW_SCRIPT_PAGE_COUNT shows up slightly more often in e10s.
Analysis was reviewed by: rvitillo
Analysis was reviewed by: rvitillo


Line 48: Line 54:
* [PARENT] JS function '''ProcessHangMonitor.reportHang''' accumulates the SLOW_SCRIPT_NOTICE_COUNT histogram (https://dxr.mozilla.org/mozilla-central/source/browser/modules/ProcessHangMonitor.jsm#403)
* [PARENT] JS function '''ProcessHangMonitor.reportHang''' accumulates the SLOW_SCRIPT_NOTICE_COUNT histogram (https://dxr.mozilla.org/mozilla-central/source/browser/modules/ProcessHangMonitor.jsm#403)
* :azhang checked that this is currently the only code path that results in SLOW_SCRIPT_NOTICE_COUNT being incremented.
* :azhang checked that this is currently the only code path that results in SLOW_SCRIPT_NOTICE_COUNT being incremented.
= Bugs =


<bugzilla>
<bugzilla>

Latest revision as of 15:26, 11 April 2016

e10s must not cause the slow-script dialog to appear more often. A/B testing on beta should indicate that the slow script dialog appears no more often than before.

RASCI

  • Responsible: azhang
  • Accountable: bsmedberg
  • Supporting:
  • Consulted:
  • Informed: cpeterson, elan, release management

Metrics

SLOW_SCRIPT_PAGE_COUNT

Summary:

  • SLOW_SCRIPT_PAGE_COUNT made it into beta46, and we've obtained data for it in the beta46-noapz and beta46-apz experiment.
  • See the latest beta46-apz experiment analysis: SLOW_SCRIPT_PAGE_COUNT still shows slightly more often on e10s than non-e10s, but the difference is much smaller than with SLOW_SCRIPT_NOTICE_COUNT.
  • We've verified that SLOW_SCRIPT_PAGE_COUNT measures the same thing across e10s and non-e10s.

Relevant info from beta46-apz slow script analysis:

Median difference in payload/histograms/SLOW_SCRIPT_PAGE_COUNT per hour is 0.12, (0.47, 0.35). The probability of this effect being purely by chance is 0.00.

Technical details:

  • SLOW_SCRIPT_PAGE_COUNT is essentially a version of SLOW_SCRIPT_NOTICE_COUNT that can be compared between e10s and non-e10s. It counts only the first slow script notice per page, which accounts for the difference in notice behaviours.
  • There should be no differences in the slow script reporting mechanism between e10s and non-e10s; 1 call to ShowSlowScriptDialog means 1 addition to the histogram regardless of e10s setting. Manual testing supports this.
  • The timeout has changed from 2 to 10 in e10s builds (see https://hg.mozilla.org/mozilla-central/rev/63680efe6d55#l35.12 for before and https://hg.mozilla.org/mozilla-central/rev/5002e9c06365#l2.13 for after). However, the value is correct on all beta versions, from 45.0b1 to 45.0b8 - this does not affect our measurements.
  • :azhang has verified that beta 46 releases with/without e10s both increment SLOW_SCRIPT_PAGE_COUNT and SLOW_SCRIPT_NOTICE_COUNT if and only if there is a slow script notification shown, and that they seem to have the exact same timeout in all cases.

Links:

Analysis results: SLOW_SCRIPT_PAGE_COUNT shows up slightly more often in e10s. Analysis was reviewed by: rvitillo

How slow script notice count works on non-e10s:

How slow script notice count works on e10s:

Bugs

Full Query
ID Summary Whiteboard Keywords Assigned to
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Retrieved from "https://wiki.allizom.org/index.php?title=Electrolysis/Release_Criteria/Slow_Script&oldid=1126965"