QA/Desktop Firefox/Releases

From MozillaWiki
Jump to navigation Jump to search

Regressions

Regression Versions Manual Coverage Automation Coverage
bug 848663 Can’t update from FF19.0 to FF19.0.1 Firefox 19+ testcase Nominated: bug 853147
bug 834667 Playing youtube video in html5 starts to leak memory Firefox 19+ testcase Nominated: bug 851502
bug 839548 Excessive memory while loading PDF thumbnail view Pending bug fix Pending bug fix Nominated: bug 851507
bug 789958 Verify update applies with root-owned files Firefox 15+ testcase n/a - not automatable
bug 789422 What's New page on Firefox update Firefox 16+ testcase nominated
bug 650355: Stop accepting md5 hash algorithm Firefox 16+ testcase nominated
Mac OSX 10.8 code-signing Firefox 17+ testcase ?
Breakpad end-to-end functionality Firefox 17+ testcase ?
Mac OSX 32-bit mode compatibility Firefox 17+ testcase ?
Mac OSX 10.5 De-support Firefox 17+ testcase ?
bug 833823 Random, unpredictable behavior with multiple YouTube embeds Firefox 20+ ? nominated
bug 787743 Private Browsing data does not cache Firefox 15+ testcase testcase

Metrics

Goal:

  • Try to identify valuable metrics in Release so we can evaluate Nightly and develop smarter strategies for Aurora
  • Value for Engineering is identifying problem areas earlier

New metrics to gather:

  • Total crashes reported for previous version effect on current version's ADIs
  • NEW vs FIXED by component and overall over time
  • Regressions and Crash bugs by component and overall over time
  • REOPENED bugs by component and overall over time
  • FIXED per component in Nightly to guide Aurora/Beta testing

Stop gather metrics on:

  • Unconfirmed vs ADIs
  • Fix verification ratio
  • Verifications vs ADIs

Visualizations

QA Metrics Bug Breakdown 20131015.png QA Metrics Unconfirmed-vs-ADI 20131015.png QA Metrics Verification Ratio 20131015.png QA Metrics Verifications-vs-ADI 20131015.png

Raw Data

Release Updates ADIs Verified Fixed Affected Wontfix Unconfirmed Total
Firefox 25 0 382 649 439 245 129 1844
Firefox 24 0 98.938 379 758 510 273 96 2016
Firefox 23 1 98.479 349 628 528 338 249 2092
Firefox 22 0 101.335 269 637 509 381 286 2082
Firefox 21 0 106.658 237 608 609 253 299 2006
Firefox 20 1 108.289 312 682 282 354 390 2020
Firefox 19 2 96.789 278 833 166 122 361 1760
Firefox 18 2 106.071 318 694 243 143 340 1738
Firefox 17 1 72.758 287 396 173 132 579 1567
Firefox 16 2 108.437 137 283 128 176 388 1112
Firefox 15 1 102.702 185 261 84 99 415 1044
Firefox 14 1 97.377 144 219 85 86 367 901
Firefox 13 1 89.853 177 169 91 74 479 990
Firefox 12 0 97.323 169 176 88 74 450 957
Firefox 11 0 94.203 164 120 88 96 426 894
Firefox 10 2 88.823 80 159 95 84 546 964

Dashboard, Spreadsheet

Crashes by Version

2013 Q4 Crashes by Version.png

Version Reported Fixed Dropped Ever Reopened Chemspills Fix Rate
Firefox 26 75 26 2 2 0 35%
Firefox 25 101 35 0 2 1 35%
Firefox 24 143 52 3 4 0 36%
Firefox 23 122 45 11 6 1 37%
Firefox 22 155 40 9 7 0 26%
Firefox 21 93 20 11 5 1 21%
Firefox 20 123 42 16 5 1 34%
Firefox 19 145 45 15 11 2 31%
Firefox 18 197 63 10 11 2 32%
Firefox 17 161 53 10 11 1 33%
Firefox 16 141 37 16 1 2 26%
Firefox 15 218 65 10 14 1 30%
Firefox 14 117 36 26 7 1 31%
Firefox 13 130 42 13 9 1 32%
Firefox 12 93 15 7 3 0 16%
Firefox 11 104 22 8 10 0 21%
Firefox 10 194 23 23 6 1 12%
Firefox 9 114 17 16 5 1 15%
Firefox 8 82 12 8 4 1 15%
Firefox 7 69 10 8 1 1 15%
Firefox 6 73 5 15 1 2 7%
Firefox 5 109 14 22 1 1 13%

Regressions by Product

Version Core Firefox Toolkit Total
Firefox 26 72 22 6 100
Firefox 25 107 23 11 141
Firefox 24 128 29 10 167
Firefox 23 141 25 4 170
Firefox 22 134 29 4 167
Firefox 21 92 16 7 115
Firefox 20 129 36 4 169
Firefox 19 154 15 8 177
Firefox 18 242 11 6 259
Firefox 17 231 25 5 261
Firefox 16 122 17 4 143
Firefox 15 191 14 8 213
Firefox 14 116 21 3 140
Firefox 13 140 24 13 177
Firefox 12 82 21 10 113
Firefox 11 67 6 9 82
Firefox 10 84 17 12 113
Firefox 9 71 20 6 97
Firefox 8 46 10 4 60
Firefox 7 47 15 5 67
Firefox 6 24 6 2 32
Firefox 5 25 4 2 31

Effect of Uplifts

Point Release Drivers

  • Firefox 23.0.1:
    • bug 901944 caused by bug 847267 which landed on mozilla-central on 2013-05-05 and rode the trains
    • bug 902532 caused by bug 857830 which landed on mozilla-central on 2013-05-07 and rode the trains
    • bug 901527 caused by bug 886886 which landed on mozilla-central on 2013-07-22 and was uplifted to mozilla-beta and mozilla-aurora after only 5 hours on Nightly
  • Firefox 20.0.1:
    • bug 857291 caused by bug 807678 which landed on mozilla-central on 2012-12-24 and rode the trains
    • bug 846848 caused by OS.File not being on QA's radar as it rode the trains

Landings per Channel

The following shows when a bug was fixed (based on the state of the status flag during the 6-week period).

Firefox Version Nightly Aurora Beta % Uplifts Point Releases
10 113 170 128 73% 2
11 90 713 283 92% 0
12 356 332 150 58% 0
13 207 316 214 72% 1
14 306 506 279 72% 1
15 340 563 381 74% 1
16 397 460 275 65% 2
17 353 563 420 75% 1
18 532 1069 545 75% 2
19 853 857 332 58% 2
20 783 644 408 57% 1
21 844 514 277 48% 0
22 550 472 247 57% 0
23 683 484 360 55% 1
24 600 494 400 60% 0
Average 467 544 313 66% 1

QA Metrics Landings per Channel 20131015.png

Conclusions & Analysis

  • Uplifts account for ~66% of all fixes going into a given release
  • All versions requiring at least one point release occurred when uplifts accounted for 55% or more of the fixes
  • All versions requiring at least two point releases occurred when uplifts accounted for ~60% or more of the fixes
  • Hypothesis: volume of landings is increasing instability (ie. likelihood of needing a follow-up release)
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Desktop_Firefox/Releases&oldid=811629"