Releases/Firefox 27/Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 339: Line 339:
  | Firefox 22.0 on Mac OSX 10.6
  | Firefox 22.0 on Mac OSX 10.6
  | Firefox 27.0b7 on Windows 8
  | Firefox 27.0b7 on Windows 8
  | {{ok|}}
  | {{done|}}
  |-
  |-
  | Firefox 27.0b7 on Mac OSX 10.7
  | Firefox 27.0b7 on Mac OSX 10.7

Revision as of 09:37, 17 January 2014

« Firefox 26 | Firefox 27 | Firefox 28 »

Firefox 27 Test Plan

Summary

Milestone Firefox 27
Status Currently in Beta
Lead Tracy Walker (irc: tracy)
2nd Anthony Hughes (irc: ashughes)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 27 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 27. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

  • 2013.09.16: [DONE] Nightly
  • 2013.10.29: [DONE] Aurora
  • 2013.12.05: [DONE] Pre-Beta feature sign-offs
  • 2013.12.12: [DONE] Beta 1
  • 2013.12.17: [DONE] Beta 2
  • skip: Beta 3
  • 2014.01.07: [DONE] Beta 4
  • 2014.01.10: [DONE] Beta 5
  • 2014.01.14: [DONE] Beta 6
  • 2014.01.17: Beta 7
  • 2014.01.21: Beta 8
  • 2014.01.24: Beta 9
  • 2014.01.28: Release Candidate
  • 2014.02.04: Final

Features

Sign-off Criteria
  • All user stories covered by manual or automated tests
  • Manual testing complete with no blocking regressions
  • Automated testing complete with no blocking regressions
  • 0 untriaged unconfirmed bugs
  • 0 untriaged unverified fixes
Status
Feature Owner Documentation Tracked Bugs Aurora Merge Sign-off Pre-beta Sign-off
Firefox Metro Juan Becerra Bug Testing [DONE] Disabled
HTTP Backend Cache Mihaela Velimiroviciu, Catalin Varga bugs Disabled NA
New First Run experience Catalin Varga 1 bug [DONE] Disabled
Add a simple tool to display reflows Petruta Rasa MDN doc 1 bug [DONE] [DONE]
SocialAPI: Allow multiple workers Petruta Rasa 1 bug NA [DONE]
Enable history swipe animations by default Juan Becerra 1 bug NA
Interface for predictive actions No need for QA owner - covered automatically 1 bug NA NA
WebGL Shader Editor No need for QA owner - covered automatically 1 bug NA NA
CSS Inheritance changes No need for QA owner - covered automatically 2 bugs NA NA

Beta

Known Issues

Full Query
ID Summary Priority Status
739043 Can't fill fillable PDF forms with PDF Viewer -- RESOLVED
810636 Poor copy & paste behavior with pdf.js P2 VERIFIED
928379 QuickTime controls bar is missing -- RESOLVED
943511 Use fallible allocation in nsSegmentedBuffer::AppendNewSegment -- VERIFIED

4 Total; 0 Open (0%); 2 Resolved (50%); 2 Verified (50%);


Regression Testing

Beginning with Fx 27, we've completely decoupled the regression testing from each beta. Instead, coverage of regression testing will take place throughout the beta cycle. The goal for Fx27 will be to finish all the regression testing prior to beta8. Breakpad will only be retested for changes landed in breakpad and/or significant changes in Socorro.

Under Test Win 7 Win 8.1 Mac 10.8 Linux
Breakpad [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1
Mac OSX code-signing N/A N/A [DONE] 27.0b1 N/A
Nvidia 3d vision [ToDo] [ToDo] [ToDo] [ToDo]
Downloads [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1
Session Restore [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1
Plug-in compatibility [DONE] 27.0b4 [DONE] 27.0b4 [DONE] 27.0b4 [DONE] 27.0b4
Mac OSX 32-bit mode N/A N/A [DONE] 27.0b4 N/A
Web compatibility [DONE] 27.0b2 [DONE] 27.0b2 [DONE] 27.0b2 [DONE] 27.0b2
Software update for root-owned files [DONE] 27.0b4 [DONE] 27.0b4 [DONE] 27.0b4 [DONE] 27.0b4
Add-on compatibility [DONE] 27.0b2 [DONE] 27.0b2 [DONE] 27.0b2 [DONE] 27.0b2
Plug-in compatibility [ToDo] [ToDo] [ToDo] [ToDo]
What's New page [DONE] 27.0b5 [DONE] 27.0b5 [DONE] 27.0b5 [DONE] 27.0b5
Guidelines
Additional Testing
  • For Fx 27.0b5 the Delicious service was tested on the following environments: Windows 7 x64 , Ubuntu 12.10 x86 and OS X 10.8. The focus of the testing was to assure proper functionality of the browser while Delicious service and sidebar are enabled.

Issues Found

List any new issues found or old issues reopened. These issues will be triaged and escalated by the release leads.

Full Query
ID Summary Priority Status
791195 Session is not restored after Firefox crash -- RESOLVED
856900 WebGL demos and games are slower on Firefox than on Chrome -- RESOLVED
943885 Investigate when "+N other downloads" should be shown instead of "Show all downloads" -- NEW
947079 Passive mixed content indicator (grey triangle) shows up on full HTTPS sites that have no mixed content -- RESOLVED
948907 Unable to scroll when selected text from textarea and outside of it P5 RESOLVED
948935 Duplicate same tool in developer tools P3 RESOLVED
948944 The download indicator is missing from the applications dock -- RESOLVED
958506 ebay.com - Button is not correctly displayed on Ebay Shopping Cart webpage -- RESOLVED
959553 Pdf with anotations doesn't print properly P3 RESOLVED
959596 PDF with annotations is incorrectly printed from Windows P3 RESOLVED

10 Total; 1 Open (10%); 9 Resolved (90%); 0 Verified (0%);


Beta 7

Build Info

Check List

  • [ON TRACK] Automation Functional tests (report)
  • [ON TRACK] Updates on betatest channel (report)
  • [ON TRACK] Updates on releasetest channel (report)
  • [ON TRACK] Updates on beta channel (report)
  • [ON TRACK] Regression testing

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Focus areas: WebRTC
Instructions
  • Use the Moztrap test as a guideline.
  • Have only one browser open on each machine at any given time
  • Have only one call running at any given time on a particular machine
  • Make sure the caller and callee are always on different machines.
  • When finding regressions, report a bug and CC Randell Jesup and Maire Reavy (they can help track down if the regressions are real or not).
    • Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment.
Caller Callee Result
Firefox 27.0b7 on Windows 7 Firefox 27.0b7 on Windows 7 [DONE]
Firefox 23.0 on Windows 7 Firefox 27.0b7 on Windows XP [ON TRACK]
Chrome Stable on Windows 7 Firefox 27.0b7 on Windows 8
Firefox 27.0b7 on Windows 7 Firefox 22.0 on Mac OSX 10.8
Firefox 27.0b7 on Windows 7 Chrome Stable on Ubuntu 12.04
Firefox 25.0 on Windows XP Firefox 27.0b7 on Windows 7 [ON TRACK]
Firefox 27.0b7 on Windows XP Firefox 24.0 on Windows XP [ON TRACK]
Firefox 27.0b7 on Windows XP Chrome Stable on Windows 8.1 [ON TRACK]
Chrome Stable on Windows XP Firefox 27.0b7 on Mac OSX 10.7
Firefox 23.0 on Windows XP Firefox 27.0b7 on Ubuntu 13.04
Chrome Stable on Windows 8.1 Firefox 27.0b7 on Windows 7 [DONE]
Firefox 27.0b7 on Windows 8 Chrome Stable on Windows XP
Firefox 22.0 on Windows 8 Firefox 27.0b7 on Windows 8.1
Firefox 27.0b7 on Windows 8.1 Firefox 26.0 on Windows 8.1
Firefox 26.0 on Windows 8 Firefox 27.0b7 on Mac OSX 10.8
Chrome Stable on Windows 8.1 Firefox 27.0b7 on Ubuntu 13.04
Firefox 27.0b7 on Mac OSX 10.9 Firefox 22.0 on Windows 7 [ON TRACK]
Chrome Stable on Mac OSX 10.8 Firefox 27.0b7 on Windows XP [ON TRACK]
Firefox 22.0 on Mac OSX 10.6 Firefox 27.0b7 on Windows 8 [DONE]
Firefox 27.0b7 on Mac OSX 10.7 Chrome Stable on Mac OSX 10.8 [DONE]
Firefox 27.0b7 on Mac OSX 10.8 Firefox 26.0 on Ubuntu 13.04 [DONE]
Firefox 27.0b7 on Ubuntu 12.04 Chrome Stable on Windows 7 [ON TRACK]
Firefox 22.0 on Ubuntu 12.04 Firefox 27.0b7 on Windows XP [ON TRACK]
Chrome Stable on Ubuntu 12.04 Firefox 27.0b7 on Windows 8 [DONE]
Firefox 27.0b7 on Ubuntu 12.04 Firefox 22.0 on Mac OSX 10.7 [DONE]
Firefox 27.0b7 on Ubuntu 12.04 Chrome Stable on Ubuntu 13.04 [ON TRACK]
Firefox 27.0b7 on Windows Vista Chrome Stable on Ubuntu 12.04 [ON TRACK]
Firefox 27.0b7 on Mac OSX 10.9 Firefox 26.0 on Windows Vista [DONE]
Firefox 27.0b7 on Windows Vista Firefox 22.0 on Widnows 7 [DONE]

Beta 6

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Regression testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)


Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: PDF Viewer, Find toolbar, App Manager
  • Platforms: Windows XP 32bit, Windows 7 64bit, Mac OSX 10.7, Ubuntu 13.10 64bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 5

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)
  • [DONE] Regression testing (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: What's New page, WebGL, Web Audio
  • Platforms: Windows 8.1 64bit, Windows 7 32bit, Mac OSX 10.8.5, Ubuntu 13.10 32bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 4

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)
  • [DONE] Regression testing (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Plugin Compatibility, Update with root owned files, Mac OSX 32-bit Mode
  • Platforms: Windows 8 64bit, Windows 7 64bit, Mac OSX 10.9, Ubuntu 12.04 32bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 2

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)
  • [DONE] Regression testing (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Secure Browsing, Add-on Compatibility, Social API
  • Platforms: Windows 8 64bit, Windows 7 64bit, Mac OSX 10.9, Ubuntu 12.04 32bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
    • Details related to "Browsing Secure Sites" test scenario can be seen in this etherpad
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 1

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Aurora

Branch Migration

Pre Sign-off Tasks

  • [DONE] Firefox branch migrated (Release Management)
  • [DONE] Mozmill Tests branch migrated (Automation Development)
  • [DONE] Updates throttled on aurora channel (Release Engineering)
  • [DONE] Updates verified throttled on aurora channel (QA)

Sign-off Tasks

  • [DONE] Mozmill Daily functional testruns pass with no failures (report)
  • [DONE] Mozmill Daily auroratest update testruns pass with no failures (report)
  • [DONE] Scope features tested and signed off for Aurora uplift
    • minimal feature work for 27

Post Sign-off Tasks

  • [DONE] Aurora sign-off email sent (QA)
  • [DONE] Updates unthrottled on aurora channel (Release Engineering)
  • [DONE] Updates verified unthrottled on aurora channel (QA)

Nightly

WebRTC

Using the Moztrap test as a guideline, verify that Firefox 27.0a1 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:

  • Have only one browser open on each machine at any given time
  • Have only one call running at any given time on a particular machine
  • Make sure the caller and callee are always on *different* machines.
  • Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
  • If you find regressions, report a bug and CC Randell Jesup and Maire Reavy; they can help track down if the regressions are real or not
    • Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
Caller Callee Result
Firefox 27.0a1 on Windows 7 Firefox 27.0a1 on Windows 7 PASS
Firefox 22.0 on Windows 7 Firefox 27.0a1 on Windows XP FAIL (1*)
Chrome Stable on Windows 7 Firefox 27.0a1 on Windows 8 PASS
Firefox 27.0a1 on Windows 7 Firefox 22.0 on Mac OSX 10.8 FAIL (3*)
Firefox 27.0a1 on Windows 7 Chrome Stable on Ubuntu 12.04 FAIL (6*)
Firefox 22.0 on Windows XP Firefox 27.0a1 on Windows 7 FAIL (1*)
Firefox 27.0a1 on Windows XP Firefox 22.0 on Windows XP FAIL (3*)
Firefox 27.0a1 on Windows XP Chrome Stable on Windows 8 PASS (4*)
Chrome Stable on Windows XP Firefox 27.0a1 on Mac OSX 10.8 PASS
Firefox 22.0 on Windows XP Firefox 27.0a1 on Ubuntu 13.04 FAIL (1*)
Chrome Stable on Windows 8 Firefox 27.0a1 on Windows 7 PASS
Firefox 27.0a1 on Windows 8 Chrome Stable on Windows XP PASS
Firefox 22.0 on Windows 8 Firefox 27.0a1 on Windows 8 PASS (5*)
Firefox 27.0a1 on Windows 8 Firefox 22.0 on Windows 8 PASS (4*)
Firefox 22.0 on Windows 8 Firefox 27.0a1 on Mac OSX 10.8 PASS (2*)
Chrome Stable on Windows 8 Firefox 27.0a1 on Ubuntu 13.04 PASS
Firefox 27.0a1 on Mac OSX 10.8 Firefox 22.0 on Windows 7 PASS
Chrome Stable on Mac OSX 10.8 Firefox 27.0a1 on Windows XP PASS (2*)
Firefox 22.0 on Mac OSX 10.8 Firefox 27.0a1 on Windows 8 PASS
Firefox 27.0a1 on Mac OSX 10.8 Chrome Stable on Mac OSX 10.8 PASS
Firefox 27.0a1 on Mac OSX 10.8 Firefox 22.0 on Ubuntu 13.04 PASS
Firefox 27.0a1 on Ubuntu 12.04 Chrome Stable on Windows 7 PASS
Firefox 22.0 on Ubuntu 12.04 Firefox 27.0a1 on Windows XP PASS
Chrome Stable on Ubuntu 12.04 Firefox 27.0a1 on Windows 8 FAIL (6*)
Firefox 27.0a1 on Ubuntu 12.04 Firefox 22.0 on Mac OSX 10.8 FAIL (3*)
Firefox 27.0a1 on Ubuntu 12.04 Chrome Stable on Ubuntu 13.04 PASS

Detailed Results:

  • (1*) Image freezes after 2 minutes or less on callee side (the caller video freezes and the audio is working)
    • not reproducible with Firefox 22 and 25.0a2
    • bug 922132
  • (2*) callee hears delayed sound (~5seconds)
    • reproducible with Firefox 22.0
  • (3*) image freezes on caller side
    • not reproducible with Firefox 22.0, 25beta2
    • reproducible with Firefox 26.0a2
    • bug 922051
  • (4*) Caller hears delayed sound
    • reproducible with Firefox 22.0
  • (5*) Callee hears delayed sound
    • reproducible with Firefox 22.0
  • (6*) Caller and callee remain in "Connecting..." state
    • reproducible with Firefox 22 and 26.0a2

Bug Triage

Query Description Remaining
QAWANTED Bugs requiring immediate investigative assistance 1
VERIFYME Bugs with fixes requiring immediate verification
FIXED Bugs with fixes which should be verified as soon as possible
REGRESSION Bugs requiring identification of the first affected build
UNCONFIRMED Bugs requiring confirmation if they are reproducible in a current build
Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Firefox_27/Test_Plan&oldid=896076"