QA/Desktop Firefox/Releases
< QA | Desktop Firefox
Regressions
| Regression | Versions | Manual Coverage | Automation Coverage |
| bug 848663 Can’t update from FF19.0 to FF19.0.1 | Firefox 19+ | testcase | Nominated: bug 853147 |
| bug 834667 Playing youtube video in html5 starts to leak memory | Firefox 19+ | testcase | Nominated: bug 851502 |
| bug 839548 Excessive memory while loading PDF thumbnail view | Pending bug fix | Pending bug fix | Nominated: bug 851507 |
| bug 789958 Verify update applies with root-owned files | Firefox 15+ | testcase | n/a - not automatable |
| bug 789422 What's New page on Firefox update | Firefox 16+ | testcase | nominated |
| bug 650355: Stop accepting md5 hash algorithm | Firefox 16+ | testcase | nominated |
| Mac OSX 10.8 code-signing | Firefox 17+ | testcase | ? |
| Breakpad end-to-end functionality | Firefox 17+ | testcase | ? |
| Mac OSX 32-bit mode compatibility | Firefox 17+ | testcase | ? |
| Mac OSX 10.5 De-support | Firefox 17+ | testcase | ? |
| bug 833823 Random, unpredictable behavior with multiple YouTube embeds | Firefox 20+ | ? | nominated |
| bug 787743 Private Browsing data does not cache | Firefox 15+ | testcase | testcase |
Metrics
Goal:
- Try to identify valuable metrics in Release so we can evaluate Nightly and develop smarter strategies for Aurora
- Value for Engineering is identifying problem areas earlier
New metrics to gather:
- Total crashes reported for previous version effect on current version's ADIs
- NEW vs FIXED by component and overall over time
- Regressions and Crash bugs by component and overall over time
- REOPENED bugs by component and overall over time
- FIXED per component in Nightly to guide Aurora/Beta testing
Stop gather metrics on:
- Unconfirmed vs ADIs
- Fix verification ratio
- Verifications vs ADIs
Visualizations
Raw Data
| Release | Updates | ADIs | Verified | Fixed | Affected | Wontfix | Unconfirmed | Total |
| Firefox 25 | 0 | 382 | 649 | 439 | 245 | 129 | 1844 | |
| Firefox 24 | 0 | 98.938 | 379 | 758 | 510 | 273 | 96 | 2016 |
| Firefox 23 | 1 | 98.479 | 349 | 628 | 528 | 338 | 249 | 2092 |
| Firefox 22 | 0 | 101.335 | 269 | 637 | 509 | 381 | 286 | 2082 |
| Firefox 21 | 0 | 106.658 | 237 | 608 | 609 | 253 | 299 | 2006 |
| Firefox 20 | 1 | 108.289 | 312 | 682 | 282 | 354 | 390 | 2020 |
| Firefox 19 | 2 | 96.789 | 278 | 833 | 166 | 122 | 361 | 1760 |
| Firefox 18 | 2 | 106.071 | 318 | 694 | 243 | 143 | 340 | 1738 |
| Firefox 17 | 1 | 72.758 | 287 | 396 | 173 | 132 | 579 | 1567 |
| Firefox 16 | 2 | 108.437 | 137 | 283 | 128 | 176 | 388 | 1112 |
| Firefox 15 | 1 | 102.702 | 185 | 261 | 84 | 99 | 415 | 1044 |
| Firefox 14 | 1 | 97.377 | 144 | 219 | 85 | 86 | 367 | 901 |
| Firefox 13 | 1 | 89.853 | 177 | 169 | 91 | 74 | 479 | 990 |
| Firefox 12 | 0 | 97.323 | 169 | 176 | 88 | 74 | 450 | 957 |
| Firefox 11 | 0 | 94.203 | 164 | 120 | 88 | 96 | 426 | 894 |
| Firefox 10 | 2 | 88.823 | 80 | 159 | 95 | 84 | 546 | 964 |
Crashes by Version
| Version | Reported | Fixed | Dropped | Ever Reopened | Chemspills | Fix Rate |
| Firefox 26 | 75 | 26 | 2 | 2 | 0 | 35% |
| Firefox 25 | 101 | 35 | 0 | 2 | 1 | 35% |
| Firefox 24 | 143 | 52 | 3 | 4 | 0 | 36% |
| Firefox 23 | 122 | 45 | 11 | 6 | 1 | 37% |
| Firefox 22 | 155 | 40 | 9 | 7 | 0 | 26% |
| Firefox 21 | 93 | 20 | 11 | 5 | 1 | 21% |
| Firefox 20 | 123 | 42 | 16 | 5 | 1 | 34% |
| Firefox 19 | 145 | 45 | 15 | 11 | 2 | 31% |
| Firefox 18 | 197 | 63 | 10 | 11 | 2 | 32% |
| Firefox 17 | 161 | 53 | 10 | 11 | 1 | 33% |
| Firefox 16 | 141 | 37 | 16 | 1 | 2 | 26% |
| Firefox 15 | 218 | 65 | 10 | 14 | 1 | 30% |
| Firefox 14 | 117 | 36 | 26 | 7 | 1 | 31% |
| Firefox 13 | 130 | 42 | 13 | 9 | 1 | 32% |
| Firefox 12 | 93 | 15 | 7 | 3 | 0 | 16% |
| Firefox 11 | 104 | 22 | 8 | 10 | 0 | 21% |
| Firefox 10 | 194 | 23 | 23 | 6 | 1 | 12% |
| Firefox 9 | 114 | 17 | 16 | 5 | 1 | 15% |
| Firefox 8 | 82 | 12 | 8 | 4 | 1 | 15% |
| Firefox 7 | 69 | 10 | 8 | 1 | 1 | 15% |
| Firefox 6 | 73 | 5 | 15 | 1 | 2 | 7% |
| Firefox 5 | 109 | 14 | 22 | 1 | 1 | 13% |
Effect of Uplifts
Point Release Drivers
- Firefox 23.0.1:
- bug 901944 caused by bug 847267 which landed on mozilla-central on 2013-05-05 and rode the trains
- bug 902532 caused by bug 857830 which landed on mozilla-central on 2013-05-07 and rode the trains
- bug 901527 caused by bug 886886 which landed on mozilla-central on 2013-07-22 and was uplifted to mozilla-beta and mozilla-aurora after only 5 hours on Nightly
- Firefox 20.0.1:
- bug 857291 caused by bug 807678 which landed on mozilla-central on 2012-12-24 and rode the trains
- bug 846848 caused by OS.File not being on QA's radar as it rode the trains
Landings per Channel
The following shows when a bug was fixed (based on the state of the status flag during the 6-week period).
| Firefox Version | Nightly | Aurora | Beta | % Uplifts | Point Releases |
| 10 | 113 | 170 | 128 | 73% | 2 |
| 11 | 90 | 713 | 283 | 92% | 0 |
| 12 | 356 | 332 | 150 | 58% | 0 |
| 13 | 207 | 316 | 214 | 72% | 1 |
| 14 | 306 | 506 | 279 | 72% | 1 |
| 15 | 340 | 563 | 381 | 74% | 1 |
| 16 | 397 | 460 | 275 | 65% | 2 |
| 17 | 353 | 563 | 420 | 75% | 1 |
| 18 | 532 | 1069 | 545 | 75% | 2 |
| 19 | 853 | 857 | 332 | 58% | 2 |
| 20 | 783 | 644 | 408 | 57% | 1 |
| 21 | 844 | 514 | 277 | 48% | 0 |
| 22 | 550 | 472 | 247 | 57% | 0 |
| 23 | 683 | 484 | 360 | 55% | 1 |
| 24 | 600 | 494 | 400 | 60% | 0 |
| Average | 467 | 544 | 313 | 66% | 1 |
Conclusions & Analysis
- Uplifts account for ~66% of all fixes going into a given release
- All versions requiring at least one point release occurred when uplifts accounted for 55% or more of the fixes
- All versions requiring at least two point releases occurred when uplifts accounted for ~60% or more of the fixes
- Hypothesis: volume of landings is increasing instability (ie. likelihood of needing a follow-up release)