MozillaWiki

QA/Desktop Firefox/Releases: Difference between revisions

Page Discussion

QA/Desktop Firefox/Releases (view source)

Revision as of 19:02, 7 February 2014

4,937 bytes removed ,  7 February 2014
no edit summary
No edit summary
 
(26 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Regressions ==
* Analysis
{| class="querytable"
** [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Effect_Of_Uplifts Effect of Uplifts] (Written as of Firefox 24)
| class="header" | Regression
** [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Metrics Metrics] (Written as of Firefox 26)
| class="header" | Versions
** [https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Regressions Regressions] (Written as of 2014-02-07)
| class="header" | Manual Coverage
* Test Plans:
| class="header" | Automation Coverage
** [https://wiki.mozilla.org/Releases/Firefox_30/Test_Plan Firefox 30]  
|-
** [https://wiki.mozilla.org/Releases/Firefox_29/Test_Plan Firefox 29]  
| {{bug|848663}} Can’t update from FF19.0 to FF19.0.1
** [https://wiki.mozilla.org/Releases/Firefox_28/Test_Plan Firefox 28]  
| Firefox 19+
** [https://wiki.mozilla.org/Releases/Firefox_27/Test_Plan Firefox 27]
| [https://moztrap.mozilla.org/manage/case/6785/ testcase]
** [https://wiki.mozilla.org/Releases/Firefox_26/Test_Plan Firefox 26]
| Nominated: {{bug|853147}}
** [https://wiki.mozilla.org/Releases/Firefox_25/Test_Plan Firefox 25]
|-
** [https://wiki.mozilla.org/Releases/Firefox_24/Test_Plan Firefox 24]
| {{bug|834667}} Playing youtube video in html5 starts to leak memory
** [https://wiki.mozilla.org/Releases/Firefox_23/Test_Plan Firefox 23]
| Firefox 19+
** [https://wiki.mozilla.org/Releases/Firefox_22/Test_Plan Firefox 22]
| [https://moztrap.mozilla.org/manage/case/6788/ testcase]
** [https://wiki.mozilla.org/Releases/Firefox_21/Test_Plan Firefox 21]
| Nominated: {{bug|851502}}
** [https://wiki.mozilla.org/Releases/Firefox_20/Test_Plan Firefox 20]
|-
** [https://wiki.mozilla.org/Releases/Firefox_19/Test_Plan Firefox 19]
| {{bug|839548}} Excessive memory while loading PDF thumbnail view
** [https://wiki.mozilla.org/Releases/Firefox_18/Test_Plan Firefox 18]
| Pending bug fix
** [https://wiki.mozilla.org/Releases/Firefox_17/Test_Plan Firefox 17]
| Pending bug fix
** [https://wiki.mozilla.org/Releases/Firefox_16/Test_Plan Firefox 16]
| Nominated: {{bug|851507}}
** [https://wiki.mozilla.org/Releases/Firefox_15/Test_Plan Firefox 15]
|-
** [https://wiki.mozilla.org/Releases/Firefox_14/Test_Plan Firefox 14]
| {{bug|789958}} Verify update applies with root-owned files
** [https://wiki.mozilla.org/Releases/Firefox_13/Test_Plan Firefox 13]
| Firefox 15+
** [https://wiki.mozilla.org/Releases/Firefox_12/Test_Plan Firefox 12]
| [https://moztrap.mozilla.org/manage/case/1372/ testcase]
** [https://wiki.mozilla.org/Releases/Firefox_11/Test_Plan Firefox 11]
| n/a - not automatable
** [https://wiki.mozilla.org/Releases/Firefox_10/Test_Plan Firefox 10]
|-
** [https://wiki.mozilla.org/Releases/Firefox_9/Test_Plan Firefox 9]
| {{bug|789422}} What's New page on Firefox update
** [https://wiki.mozilla.org/Releases/Firefox_8/Test_Plan Firefox 8]
| Firefox 16+
** [https://wiki.mozilla.org/Releases/Firefox_7/Test_Plan Firefox 7]
| [https://moztrap.mozilla.org/manage/case/1371/ testcase]
** [https://wiki.mozilla.org/Releases/Firefox_6/Test_Plan Firefox 6]
| nominated
** [https://wiki.mozilla.org/Releases/Firefox_5/Test_Plan Firefox 5]
|-
| {{bug|650355}}: Stop accepting md5 hash algorithm
| Firefox 16+
| [https://moztrap.mozilla.org/manage/case/1292/ testcase]
| nominated
|-
| Mac OSX 10.8 code-signing
| Firefox 17+
| [https://moztrap.mozilla.org/manage/case/2524/ testcase]
| ?
|-
| Breakpad end-to-end functionality
| Firefox 17+
| [https://moztrap.mozilla.org/manage/case/2525/ testcase]
| ?
|-
| Mac OSX 32-bit mode compatibility
| Firefox 17+
| [https://moztrap.mozilla.org/manage/case/2526/ testcase]
| ?
|-
| Mac OSX 10.5 De-support
| Firefox 17+
| [https://moztrap.mozilla.org/manage/case/2527/ testcase]
| ?
|-
| {{bug|833823}} Random, unpredictable behavior with multiple YouTube embeds
| Firefox 20+
| ?
| nominated
|-
| {{bug|787743}} Private Browsing data does not cache
| Firefox 15+
| [https://moztrap.mozilla.org/manage/case/1529/ testcase]
| [http://hg.mozilla.org/qa/mozmill-tests/file/tip/tests/functional/testPrivateBrowsing/testAboutCache.js testcase]
|}
 
== Metrics ==
Goal:
* Try to identify valuable metrics in Release so we can evaluate Nightly and develop smarter strategies for Aurora
* Value for Engineering is identifying problem areas earlier
 
New metrics to gather:
* Total crashes reported for previous version effect on current version's ADIs
* NEW vs FIXED by component and overall over time
* Regressions and Crash bugs by component and overall over time
* REOPENED bugs by component and overall over time
* FIXED per component in Nightly to guide Aurora/Beta testing
 
Stop gather metrics on:
* Unconfirmed vs ADIs
* Fix verification ratio
* Verifications vs ADIs
 
=== Visualizations ===
[[Image:QA_Metrics_Bug_Breakdown_20131015.png]]
[[Image:QA_Metrics_Unconfirmed-vs-ADI_20131015.png]]
[[Image:QA_Metrics_Verification_Ratio_20131015.png]]
[[Image:QA_Metrics_Verifications-vs-ADI_20131015.png]]
 
=== Raw Data ===
{| class="querytable"
| class="header" style="width:10%" | Release
| class="header" style="width:10%" | Updates
| class="header" style="width:10%" | ADIs
| class="header" style="width:10%" | Verified
| class="header" style="width:10%" | Fixed
| class="header" style="width:10%" | Affected
| class="header" style="width:10%" | Wontfix
| class="header" style="width:10%" | Unconfirmed
| class="header" style="width:10%" | Total
|-
| Firefox 25
| 0
| 
| 382
| 649
| 439
| 245
| 129
| 1844
|-
| Firefox 24
| 0
| 98.938
| 379
| 758
| 510
| 273
| 96
| 2016
|-
| Firefox 23
| 1
| 98.479
| 349
| 628
| 528
| 338
| 249
| 2092
|-
| Firefox 22
| 0
| 101.335
| 269
| 637
| 509
| 381
| 286
| 2082
|-
| Firefox 21
| 0
| 106.658
| 237
| 608
| 609
| 253
| 299
| 2006
|-
| Firefox 20
| 1
| 108.289
| 312
| 682
| 282
| 354
| 390
| 2020
|-
| Firefox 19
| 2
| 96.789
| 278
| 833
| 166
| 122
| 361
| 1760
|-
| Firefox 18
| 2
| 106.071
| 318
| 694
| 243
| 143
| 340
| 1738
|-
| Firefox 17
| 1
| 72.758
| 287
| 396
| 173
| 132
| 579
| 1567
|-
| Firefox 16
| 2
| 108.437
| 137
| 283
| 128
| 176
| 388
| 1112
|-
| Firefox 15
| 1
| 102.702
| 185
| 261
| 84
| 99
| 415
| 1044
|-
| Firefox 14
| 1
| 97.377
| 144
| 219
| 85
| 86
| 367
| 901
|-
| Firefox 13
| 1
| 89.853
| 177
| 169
| 91
| 74
| 479
| 990
|-
| Firefox 12
| 0
| 97.323
| 169
| 176
| 88
| 74
| 450
| 957
|-
| Firefox 11
| 0
| 94.203
| 164
| 120
| 88
| 96
| 426
| 894
|-
| Firefox 10
| 2
| 88.823
| 80
| 159
| 95
| 84
| 546
| 964
|}
''[http://dataviz.mozilla.org/views/DesktopOverview/Overview Dashboard]'', ''[https://docs.google.com/spreadsheet/ccc?key=0AmkRt0ylPb8zdGRjTXpkdlRrTUhtOEp3OXB0LWcxeFE#gid=13 Spreadsheet]''
 
=== Crashes by Version ===
 
== Effect of Uplifts ==
=== Point Release Drivers ===
* Firefox 23.0.1:
** {{bug|901944}} caused by {{bug|847267}} which landed on mozilla-central on 2013-05-05 and rode the trains
** {{bug|902532}} caused by {{bug|857830}} which landed on mozilla-central on 2013-05-07 and rode the trains
** {{bug|901527}} caused by {{bug|886886}} which landed on mozilla-central on 2013-07-22 and was uplifted to mozilla-beta and mozilla-aurora after only 5 hours on Nightly
* Firefox 20.0.1:
** {{bug|857291}} caused by {{bug|807678}} which landed on mozilla-central on 2012-12-24 and rode the trains
** {{bug|846848}} caused by OS.File not being on QA's radar as it rode the trains
 
=== Landings per Channel ===
The following shows when a bug was fixed (based on the state of the status flag during the 6-week period).
 
{| class="querytable" style="width:60%"
| class="header" style="width:10%" | Firefox Version
| class="header" style="width:10%" | Nightly
| class="header" style="width:10%" | Aurora
| class="header" style="width:10%" | Beta
| class="header" style="width:10%" | % Uplifts
| class="header" style="width:10%" | Point Releases
|-
| 10
| 113
| 170
| 128
| 73%
| 2
|-
| 11
| 90
| 713
| 283
| 92%
| 0
|-
| 12
| 356
| 332
| 150
| 58%
| 0
|-
| 13
| 207
| 316
| 214
| 72%
| 1
|-
| 14
| 306
| 506
| 279
| 72%
| 1
|-
| 15
| 340
| 563
| 381
| 74%
| 1
|-
| 16
| 397
| 460
| 275
| 65%
| 2
|-
| 17
| 353
| 563
| 420
| 75%
| 1
|-
| 18
| 532
| 1069
| 545
| 75%
| 2
|-
| 19
| 853
| 857
| 332
| 58%
| 2
|-
| 20
| 783
| 644
| 408
| 57%
| 1
|-
| 21
| 844
| 514
| 277
| 48%
| 0
|-
| 22
| 550
| 472
| 247
| 57%
| 0
|-
| 23
| 683
| 484
| 360
| 55%
| 1
|-
| 24
| 600
| 494
| 400
| 60%
| 0
|-
| '''Average'''
| '''467'''
| '''544'''
| '''313'''
| '''66%'''
| '''1'''
|}
 
[[Image:QA_Metrics_Landings_per_Channel_20131015.png]]
 
=== Conclusions & Analysis ===
* Uplifts account for ~66% of all fixes going into a given release
* All versions requiring at least one point release occurred when uplifts accounted for 55% or more of the fixes
* All versions requiring at least two point releases occurred when uplifts accounted for ~60% or more of the fixes
* ''Hypothesis'': volume of landings is increasing instability (ie. likelihood of needing a follow-up release)
Ashughes
Confirmed users
14,525

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/811305...920080"