Releases/Firefox 47/Test Plan

Summary

This page is to track testing of Firefox 47 from mozilla-central (Nightly) through to mozilla-release. This is intended to be a minimal set of information to help Firefox QA understand and communicate the quality of upcoming releases. Our current walkthrough of the release and QA process might be changing due to recent reorgs.

Schedule

Please consult the Rapid Release Calendar for more information.

Milestone	Date	Checks
Nightly	2016-01-25	Conduct daily stability and bug triage, ensure features are owned
Aurora Migration	2016-03-07	Conduct testing to ensure Firefox 47 Aurora builds are okay, sign off for updates on Friday
Aurora Updates enabled	2016-03-11	Continue daily stability and bug triage, ensure features are tested
Beta Uplift	2016-04-18	Review testing and sign off every beta release
Release	2016-05-31	Review tests and sign off release

Meetings

Meeting	Purpose	When	Vidyo Room	IRC Backchannel
Channel Status	Raise quality concerns with the Release Management team	10:00 Pacific on Tuesday & Thursday	ReleaseCoordination	#planning
Firefox QA	Meet the Desktop QA team and hear what they're working on	8:30am Pacific on Wednesdays	DesktopQA	#qa
QA Team	Meet the entire QA team and hear what they're working on!	1:30am Pacific on Wednesdays	QA	#qa

Nightly

During the 6 weeks that Firefox 47 is on the Nightly channel, QA will focus on helping with the Firefox iteration planning, as well as with crash reporting and triaging newly reported bugs. The earlier we can get these bug reports into shape for developers to prioritize them and work on them, the more likely they'll be fixed by release in April.

Reference: https://wiki.mozilla.org/QA/Desktop_Firefox/Walkthroughs/Release_Coordination#Nightly

Questions for workweek

• Each topcrash should have a developer assigned, a QA contact, and tracking + or -.
• We should decide on a threshold for "crisis" level crash spikes/crash rates. (Different for startup, plugin, crashes or specific platforms)
• What do we want to do with the iteration process. Continue assigning QA contact, verification + or -?
• We should encourage the use of "feature" or verification needed keywords/flags.
• What, if any, conditions would actually block or delay the move to aurora?

End of cycle status

• Total bugs filed affecting 47:
• Bugs fixed / verified on 47:
• Conditional signoff for merge: (date, who signed off)
• QE signoff:

Developer Edition (Aurora)

Continue triaging incoming bugs with community help. Test e10s issues.

During the 47 Developer Edition cycle we will be working to help determine the state of the Electrolysis (e10s) project. e10s will be ready to move to Beta if stability and fixes are up to an acceptable quality level.

• File crashes, escalate bad crashes.
• Track feature milestones (for example the e10s tracking flags.)
• Should have testing plans for features that we know may be turned off /on conditionally.
• link to release notes. review & update.

Reference (and migration / signoff plan: https://wiki.mozilla.org/QA/Desktop_Firefox/Walkthroughs/Release_Coordination#Aurora)

Beta

During the Beta cycle, there are at least 9 beta releases. Communication with the release coordination, release engineering, and automation teams is key.

QA analyzes the results of functional, remote, and update tests running on jenkins. We create config files for the update tests, check their validity on mciconf and run them on at least two channels for each beta release cycle. A big part of this work currently is going through the test results to figure out why test failures occurred (infrastructure, tests themselves, actual Firefox bug).

Pivotal bugs may need verification during each beta cycle.

• link to release notes. review & update.
• crash evaluation/escalation.
• feature evaluation.

QA Activities

QA wanted

• QA Needed Bugs

The following tracks bugs which have requests for urgent QA assistance.

• Choose a bug to test from the list below
• Read it and make sure you understand what information is needed (this may be steps or URLs to reproduce, a regression range, or something else altogether)
• Use the latest build of this Firefox version and test until you've discovered the information necessary. Ask for help if you need it.
• Report this information to the bug and remove the qawanted keyword

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Stability

This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.

• Ensure new and explosive crashes have actionable bug reports which are in progress
• Top Crashes - ensure the highest ranked crashes have actionable bug reports which are in progress
• Untriaged Bugs - ensure bugs are actionable and in the correct component
• topcrash bugs

Unconfirmed Bug Triage

The following tracks triage and testing of incoming, unconfirmed bug reports.

• Choose a bug to test from the list below
• Use the latest Nightly build to confirm the bug reproduces
• Flag with reporter with needinfo if you cannot reproduce and need more information
• Set the status to NEW and ensure the bug is in the appropriate component if you can reproduce the bug (provide as much detail about your testing as possible)
• If it's a Firefox bug you may want to add the firefox-backlog=? flag to it.

Queries:

• All Unconfirmed Bugs
  • https://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&version=47%20Branch
• Unchanged Since One Week
  • These bugs may need extra attention.
  • https://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&version=47%20Branch&chfieldto=-1w
• Unchanged Since One Month
  • These are likely to be confusing or non-reproducible issues. We should make decisions to either close these or tag these unactionable, and filter them out
  • https://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&version=47%20Branch&chfieldto=-1m

Verification Needed Bugs

• Review the bugs below in the Pending Triage section and select a bug you want to test
• Make sure you understand the bug before you begin testing it
• Test on a previously known broken build to make sure you can reproduce the bug
• Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
• Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
• Reopen the bug if it still reproduces

Iteration Development

Firefox development during Nightly works on two-week iterations as of 2014. This section tracks the bugs selected for development in these iterations. Please consult the Iteration Dev Walkthrough and the Firefox/IterativeDevelopment page for more information on this process.

• Ensure all bugs are marked qe-verify:+ or -
• Ensure all qe-verify:+ bugs have a QA Contact (The Feature Owner page may help here)
• Ensure all selected bugs are VERIFIED before the end of the iteration
• Anything incomplete at the end of the iteration will be automatically carried over to the next iteration

Queries:

• Dashboard showing current active iteration: https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/qa/
• Iteration 47.1, all bugs: https://bugzilla.mozilla.org/buglist.cgi?f1=OP&j1=OR&f2=cf_fx_iteration&o2=substring&v2=47.1&f3=status_whiteboard&o3=substring&v3=s%3D47.1&f4=CP
• Iteration 47.2, all bugs: https://bugzilla.mozilla.org/buglist.cgi?f1=OP&j1=OR&f2=cf_fx_iteration&o2=substring&v2=47.2&f3=status_whiteboard&o3=substring&v3=s%3D47.2&f4=CP
• Iteration 47.3, all bugs: https://bugzilla.mozilla.org/buglist.cgi?f1=OP&j1=OR&f2=cf_fx_iteration&o2=substring&v2=47.3&f3=status_whiteboard&o3=substring&v3=s%3D47.3&f4=CP

Milestones

Aurora Migration

Post-merge Work (as soon as updates are disabled):

• [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the auroratest channel (before first Aurora builds are generated, to make sure we don't get any red on aurora update tests)

Pre-sign-off Checks:

• [DONE] Features - all scoped features are signed off as ready for Aurora by owners. (If you haven't heard anything to the contrary, that is probably true, but make sure we don't know of blocking issues.)
• [DONE] Bugs - document and review any reported bugs and issues, nominate them for tracking if serious (i.e. make sure all bugs needed to be tracked have at least a request for that up).
• [DONE] Stability - review crash stats before sign-off to raise any red flags.
• [DONE] L10n - contact L10n drivers (usually :Pike) to see if there are any known localization issues.

Post-sign-off Checks (as soon as updates are enabled again):

• [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel

Betas

Beta 1

Build Information

• Build 2: 47.0b1-build2 | 5bbf2e7c2fc6 | (ID: 20160425205003)

Testing

• Manual - Firefox 47 Beta 1
  • [DONE] Smoke Testing with focus on YouTube embedded rewrite, Synced tabs sidebar, Per-monitor DPI.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Beta 2

Build Information

• Build 1: 47.0b2-build1 | 6f82d30fe05e | (ID: 20160502152141)

Testing

• Manual - Firefox 47 Beta 2
  • [DONE] Smoke Testing with focus on WebGL, Crash reporter, Accessibility (Keyboard Shortcuts).
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Beta 3

Build Information

• Build 1: 47.0b3-build1 | 609000bcc112 | (ID: 20160505125249)

Testing

• Manual - Firefox 47 Beta 3
  • [DONE] Smoke Testing with focus on Image Support, Audio Compatibility, PDF Compatibility.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Beta 4

Build Information

• Build 1: 47.0b4-build1 | 2991f214d4f4 | (ID: 20160509171155)

Testing

• Manual - Firefox 47 Beta 4
  • [DONE] Smoke Testing with focus on Video Compatibility, Web Compatibility, Toolbar Customization.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Beta 5

Build Information

• Build 1: 47.0b5-build1 | 93a53170dedf | (ID: 20160512003946)

Testing

• Manual - Firefox 47 Beta 5
  • [DONE] Smoke Testing with focus on Migration, OS X Signing, OS X 32-bit Mode, and Printing.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Beta 6

Build Information

• Build 1: 47.0b6-build1 | 7d1f3450acc4 | (ID: 20160516123243)

Testing

• Manual - Firefox 47 Beta 6
  • [DONE] Smoke Testing with focus on Plug-in Compatibility, Session Restore, Downloads, Safe Browsing, Bookmarks.
    • The build is [GREEN] so far, but there's one pending Safe Browsing test that requires a 14 hrs wait time. We'll follow up on it tomorrow.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Beta 7

Build Information

• Build 1: 47.0b7-build1 | 0723a0212f5e | (ID: 20160518173344)

Testing

• Manual - Firefox 47 Beta 7
  • [DONE] Smoke Testing with focus on Add-on Compatibility, Search, and Themes.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Beta 8

Build Information

• Build 1: 47.0b8-build1 | cb27eacbe04a | (ID: 20160523113146)

Testing

• Manual - Firefox 47 Beta 8
  • [DONE] Smoke Testing with focus on Sync, Video Compatibility, and Default Browser setting.
    • Open H264 Regression Testing is also covered in Beta 8, but it is not part of the sign-off
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Beta 9

Build Information

• Build 1: 47.0b9-build1 | 2ee4473c729a | (ID: 20160526140250)

Testing

• Manual - Firefox 47 Beta 9
  • [DONE] Smoke Testing with focus on Web Compatibility, and Video Compatibility.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

Release Candidates

47.0 RC

Build Information

• Build 1: 47.0RC | fe0d30de989d | (ID: 20160531183335)

Testing

• Manual - Firefox 47.0 RC
  • [DONE] Smoke Testing with focus on Web Compatibility, Graphics and Per-monitor DPI.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • This is a candidate for the 47.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

47.0 RC2

Build Information

• Build 2: 47.0RC2 | 8815c6471061 | (ID: 20160602131235)

Testing

• Manual - Firefox 47.0 RC2
  • [DONE] Smoke Testing with focus on Web Compatibility.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • This is a candidate for the 47.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)

47.0 RC3

Build Information

• Build 3: 47.0RC3 | b0310cb90fd0 | (ID: 20160604131506)

Testing

• Manual - Firefox 47.0 RC3
  • [DONE] Smoke Testing with focus on Video Compatibility.
  • [DONE] Bug verification with focus on the driver for this build - bug 1278005.
• Automated (configs for ondemand)
  • This is a candidate for the 47.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
  • [DONE] Updates on beta-cdntest channel (report)
  • [DONE] Updates on beta channel (report)
  • [DONE] Updates on release-cdntest channel (report)
  • [DONE] Updates on release channel (report)

Release 47.0

Build Information

• See Release Candidate Build 3

Testing

• Update Testing - see Release Candidate Build 3
• Throttle Testing (see Walkthrough for details, and detailed Test Instructions for running them)
  • Used commands:
    • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
    • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/WINNT_x86_64-msvc/en-US/release/Windows_NT%2010.0/default/default/update.xml
    • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2013.4.0/default/default/update.xml
    • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/Linux_x86-gcc3/en-US/release/Linux%203.13.0-45-generic%20%28GTK%202.24.23%29/default/default/update.xml
    • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/Linux_x86_64-gcc3/en-US/release/Linux%203.16.0-30-generic%20%28GTK%202.24.25%29/default/default/update.xml
  • Results
    • After release push (expected to be @ 25%), 2016-06-07: Win32: 24% - Win64: 23% - Mac: 25% - Linux32: 24% - Linux64: 24%
    • After disabling updates (expected to be @ 0%), 2016-06-08: Win32: 0% - Win64: 0% - Mac: 0% - Linux32: 0% - Linux64: 0%
    • After fully enabling updates (expected to be @ 100%): 2016-06-10: Win32: 100% - Win64: 100% - Mac: 100% - Linux32: 100% - Linux64: 100%

47.0.1 Build 1

Build Information

• Build 1: 47.0.1-build1 | 7f5abf95991b | (ID: 20160623154057)
• Manual - Firefox 47.0.1
  • [DONE] Regression Testing with focus on Web Compatibility.
  • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on release-localtest channel (report)
  • [DONE] Updates on release-cdntest channel (report)
  • [DONE] Updates on release channel (report)
  • Throttle Testing (see Walkthrough for details)
    • Used commands:
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/WINNT_x86_64-msvc/en-US/release/Windows_NT%2010.0/default/default/update.xml
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2013.4.0/default/default/update.xml
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/Linux_x86-gcc3/en-US/release/Linux%203.13.0-45-generic%20%28GTK%202.24.23%29/default/default/update.xml
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/Linux_x86_64-gcc3/en-US/release/Linux%203.16.0-30-generic%20%28GTK%202.24.25%29/default/default/update.xml
    • Results
    • After release push (expected to be @ 25%), 2016-06-28: Win32: % - Win64: % - Mac: % - Linux32: % - Linux64: %
    • After disabling updates (expected to be @ 0%), 2016-06-29: Win32: % - Win64: % - Mac: % - Linux32: % - Linux64: %
    • After fully enabling updates (expected to be @ 100%): 2016-07-01: Win32: % - Win64: % - Mac: % - Linux32: % - Linux64: %
    • NOTE: throttle tests do not apply to this release as we released at 100% for versions < 47.0, and 0% for 47.0 (except for forced updates) - this was confirmed by ondemand update tests on the release channel (for forced updates), and throttling tests which were at 0% (using the throttling commands above)

47.0.2 Build 1

Build Information

• Build 1: 47.0.2-build1 | 3dc66a6a33a0 | (ID: 20161027172622)
• Manual - Firefox 47.0.2
  • [DONE] Sanity Testing with focus on: general Web Compatibility.
    • Detailed test results: in TestRail (requires access rights) or as a pdf.
  • [DONE] Verification of driver bug 1305787
• Automated (configs for ondemand)
  • [DONE] Updates on release-localtest channel (report)
    • Also tested manually that:
      • Fx < 47.0.2 with the hotfix installed and (websense) get updated to 47.0.2
      • Fx < 47.0.2 with the hotfix installed and (nowebsense) get updated to 48.0.2 and then 49.0.2
  • [DONE] Updates on release-cdntest channel (report)
  • [NOT STARTED] Updates on release channel (report)
  • Throttle Testing (see Walkthrough for details)
    • N/A, as Build 2 was generated

47.0.2 Build 2

Build Information

• Build 2: 47.0.2-build2 | d7f98b92fdec | (ID: 20161031133903)
• Manual - Firefox 47.0.2
  • [DONE] Sanity Testing with focus on: general Web Compatibility.
    • Detailed test results: in TestRail (requires access rights) or as a pdf.
• Automated (configs for ondemand)
  • [DONE] Updates on release-localtest channel (report)
    • Also tested manually that:
      • Fx < 47.0.2 with the hotfix installed and (websense) get updated to 47.0.2
      • Fx < 47.0.2 with the hotfix installed and (nowebsense) get updated to 48.0.2 and then 49.0.2
  • [DONE] Updates on release-cdntest channel (report)
  • [DONE] Updates on release channel
    • Automated tests failures - (report)
    • Manual tests results
  • Throttle Testing (see Walkthrough for details)
    • Used commands:
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/WINNT_x86_64-msvc/en-US/release/Windows_NT%2010.0/default/default/update.xml
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2013.4.0/default/default/update.xml
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/Linux_x86-gcc3/en-US/release/Linux%203.13.0-45-generic%20%28GTK%202.24.23%29/default/default/update.xml
      • ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/Linux_x86_64-gcc3/en-US/release/Linux%203.16.0-30-generic%20%28GTK%202.24.25%29/default/default/update.xml
    • Results
    • After release push (expected to be @ 25%), 2016-10-31: N/A - Releasing directly to 100%
    • After disabling updates (expected to be @ 0%), 2016-10-31: N/A - Releasing directly to 100%
    • After fully enabling updates (expected to be @ 100%): 2016-11-04: Win32: 100% - Win64: 100% - Mac: 100% - Linux32: 100% - Linux64: 100%