QA/Firefox3/TestPlan

« QA/Firefox3

Top Critical issues for QA

This is a working list that QA is using to request be fixed or completed for the Upcoming Fx3 Milestones. This list of items is NOT the same as what the Fx3 Development team already is tracking.

First, Refer to dev-tracking bugs at: [M9 firefox]|[M9 platform]

If you have untracked issues in these areas, please enter them here:

• Design still incomplete?
• Major bugs blocking your feature that should be addressed?
• Major bugs blocking overall regression of the trunk that should be fixed before shipping?

Critical For Beta 1

• QA Relnotes Tracking

• UPDATE - Verified and fixed in hourly -Bug 398446 - Can't log in to chase.com account. Has regression range but no testcase.

• UPDATE: verified and fixed from bug 401687 - Bug 402211 - a crash that was just introduced in 2007110104 build. Sorocco indicates 250 total crashes today so far.

• UPDATE: verified fixed - tomcat -Bug 389131 – Reporter should move away from SOAP (Reporter fails to send reports on trunk) Bug says resolved, but still broken.

• UPDATE: issue resolved and verified fixed - tomcat - Bug 401053 – Hang initializing scriptable Java applet. Fixed, awaiting checkin and verification.

Must fix by Beta 2

• bug 392919 - Breakpad fails with error while loading shared libraries: libcurl.so.3. Breakpad System is broken on Fedora 8
• Bug 401301 - firefox.exe process still around after all windows closed Still investigating firefox hang process. Hard to repro, but working with accessibility.
• Bug 401064 – SnowFX applet requires a click to display. Affects some java plugin sites, but not all. Needs to be fixed, but doesnt block java testing.
• UPDATE - Fixed and Verified on trunk. -Bug 399238 and Bug 397428. Google maps prints horribly on windows and mac.
• Critical Printing Bugs - These are printing crash bugs from printing. Bugs listed in this query are marked in Trunk, and have been confirmed by QA members.
  • In general, QA doesn't yet have an idea what "improve printing" means for this release; is it a metric-based, top-site notion, or something that includes some subset of prior-known, broken-printing pages, or...?
    • - stephend
• Bug 324857 – MathML all screwed up in Cairo builds. Blocking MathML usage
• Addons Manager UI has a spec, but no dev resource or timeline to work on.
  • UPDATE - (beltzner) publish the plan soon.
• Larry UI still not completed? Affects Security/Privacy feature testing.
  • DISCUSSION - (beltzner) all items are complete. Stephend, please comment on what you think is still missing here. Also include bug #'s.
    • Sure thing; Larry is a moving target, which is what I told Tony when he asked how testing is going on it (it's actually QA-owned by Juan/Carsten, BTW). I point to bug 402260, and I really don't see this as a huge issue (maybe I'm misunderstanding the intent of this page?) The other issue is that there really isn't a UI spec, so QA has a hard time knowing against what to test.
    • My worry is that small examples like this are all over the application, and I don't want QA to miss something because we don't understand which features from some round of redesign/tweak are to be included; that would be bad for all. Testcase writing takes a formidable amount of time for some really complex features, since it incorporates time spent chasing down finality of UI/behavior (which is our job, after all, so no complaints!). Just sayin'
      • - stephend
• UPDATE: fixed in b1 bug 402210 Handling SSL sites with a domain mismatch. (eg. https://versign.com instead https://www.verisign.com) needs some notes for Firefox 2 User why we have changed the SSL handling, since its different to FX2. In Firefox 2 when you was surfing to https://verisign.com instead of https://www.verisign.com you got the Domain Missmatch Error, but you could still use the site. In Firefox 3 Trunk we block this site and give users no expection button right now (or a hint to go to https://www....), this can maybe confuse users that come from Firefox 2.
• Accesibility: bug 393398 Add Bookmark Dialog Inaccessible: Needs to be classified as a dialog or a window in order to work properly with most AT. Needs testing.
• Accessibility: Bookmark/Places Panel not Accessible, need menu items to duplicate toolbar entries.
  • DISCUSSION - beltzner mentions there is work being done already. TimK, please comment on what specifics are blocking.
    • There are equivalence in the Bookmarks menu to the toolbar buttons--for some reason they weren't accessible from my profile. However, there are sec508 concerns with the Places Organizer contained in bug 400703 and bug 402104 which will require more extensive testing and should be resolved by beta 2, if at all possible. -TimK

Must Fix by Release Candidate

• Mac Focus issues still a problem; bug 354768.
• Accessibility: Download Manager, being redesigned; would like the UI solidified earlier rather than later.
  • DISCUSSION - beltzner mentions there is work being done already. TimK, please comment on what specifics are blocking.
  • bug 393248 The info button currently brings up a popup that is classed as a menu, which confuses AT. Also the search box has no accessible label. -TimK

• I'd like to know in what form of autocomplete the final Firefox 3 location bar will/should take; again, I understand there are ongoing experiments in order to get the best fit--across the board in the application--I just want everyone to be aware that churn really takes a toll on solidifying testcase-writing/execution. (Boy, I really sound whiny, no? I honestly just want us all to be on the same page, QA feels really, really behind in most areas, testing-wise, but perhaps that's just my perception...)
  • - stephend

RESOLVED FIXED bugs, needing more information or testcases

• Bug 375750 - PDFs generated by the trunk are huge. The patch seems to make the size smaller, but the size is still not in line with what the branch size is
• Bug 402000 - Cannot be verified according to dev

Overview

Firefox 3 QA Test Strategy

The goal of this Overall Test Plan is to lay out our testing strategy, communicate it to others, and track the status.

Major QA Activities:

• Litmus Cleanup
• Litmus Triage Team
• New feature testing - see below
• L10n testing - TBD
• Milestone testing - TBD

Firefox 3 Central

Requirements and Ownership

• Fx3 Product Requirements
• Fx3 Dev team up-to-date feature tracking status

Archives:

• Firefox 3 TestPlan Tracking page (Google Spreadsheet)
• Fx 3 PRD Wiki page

Test Areas

Feature Focused Areas

Key: L = Large (3-6 weeks) M = Medium (2-4 weeks) S = Small (1-2 weeks)

Test Plan Template

Feature Area and Spec	Test Plan Location	QA Contact	Size	Description
Addons	Addons Test Plan	tchung (new Addons Manager UI, Extensions), stephend (Unified Plugin Manager)	M	Add-on management system, plugin management system, localized user-facing addon text, configuration, updates, Installation, Discovery, FUEL 0.2
Content Handling	Microformat Test Plan Web Application Test Plan Download Manager Test Plan	ctalbert, stephend (Download Manager)	L	microformats, Content Type Processing, download manager, MIME type support, User Interface, Media Plugins (needs definition), firebug (at risk)
Partner Distribution	TBD	tracy, ctalbert	M	High-Touch/Low-Touch distros, regression, updates, test automation
OS Platform Integration	Vista Test Plan	Tomcat, marcia, ss (breakpad)	M	Vista Support (At Risk), MSI bundling, Airbag / Breakpad, parental controls (At Risk), Notification, Vista Installer
Gecko Platform Integration	Gecko Platform Test Plan	martijn, marcia	M	Cairo graphics, nxTextFrame migration to Thebes, Reflow Branch, Caret painting rewrite, Cross-domain XMLHttpRequest (Cocoa widgets covered under Mac Test Plan)
Mac OS Platform Integration (includes Gecko)	Mac OS Integration Test Plan Leopard Test Plan	Tomcat, marcia	M	Leopard support, Cocoa implementation, widgets, form controls, Growl Integration on Mac, Mac Installer
Places	Places Test Plan	Tracy, ABillings	L	Bookmarks, History Indexing, Data Management, User Interface, Extensibility, Performance
Search / Tabbed Browsing / Visual Refresh	Search, Tabs, Visual Refresh Test Plan Tab Switching Test	Tomcat (search), Marcia (tabbed browsing)	S	User Interfacing, Animations, Data management
Security, Privacy / Password, Identity	Security / Privacy Test Plan Password Manager Testplan	Tomcat, juanb	L	password management, Phishing protection, Private web browsing, Blacklisting (At Risk), password Manager, Password Security
User Support	TBD	Tomcat, ss	S	Help content, Reporter, feedback, Customization options Now owned by SUMO team - David Tensor.
L10N Testing	L10N Test Plan	ctalbert, tchung	S	test automation, new locales, string lengths, bookmarks, rss, search plugins
UI Accessibility	UI Accessibility Test Plan	TimK	S	UI accessibility convention testing
Major Update Tests (2.0.x -> FF3)	TBD	tchung	S	update verification from 2.0.x builds to 3.0b1+. includes upgrades/downgrades/sidegrade cases

Regression Areas

• Top Web accessibility Tests
  • Plugins (eg. Flash, java, AJAX)
  • Extensions (eg. Firebug, NoScript, Fotofox)
  • Themes (eg. Vista, Joga, Simpsons)
  • Top Site Tests (eg. Alexa top 50)
  • Financial Sites (eg. banks, online trading)
• Accessibility Tests
• User Performance Testing
• Major / Minor Update Testing
• L10N Testing (may coincide with Feature area)
• Distribution Testing (may coincide with Feature area)

Ownership Needed Litmus Test Case Areas

These are Litmus test case areas that are currently unowned.

• RSS->marcia
• Printing -> Tomcat, tchung
• Location Bar -> Stephend
• Find -> TBD
• Customize Toolbar -> TBD
• Mac Leopard -> marcia

Tiger Team Testing

Daily smoketests are ran against nightly trunk builds. Please see marcia for more information.

Link to Schedule and Assignments

Extensions

For Firefox 2 we had Bug 353206 as extension Tracking bug.

The following extensions were never fixed from their extension compatibly authors to be able to install in Firefox 2.

• Slim Extension List [1]
• Bookmarks Synchronizer 3 [2]
• AutoMarks [3]
• Extended Link Properties [4]
• Snapper [5]
• Cacheout[6]
• Resize Search Box [7]
• superT 0.7.9.1 https://addons.mozilla.org/en-US/firefox/addon/2224
• Reveal extension [8]

Top Crash reports

The main report is the easiest to view.

Note that of those in that list, only a few seem to be active lately. I filed bug 402661 for the crash in RtlpCoalesceFreeBlocks / XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode). I still need to file a bug for the crash in CallGetService(nsID const&, nsID const&, void**). Neither of these should be blocking beta given their (relative) infrequency.

Banking and Financial Sites

For Firefox 3, we’re asking community members and MoCo QA to state which banks and financial institutions they're willing to test. Please add your name to the list.

Flash Beta plugin Testing

• http://weblogs.macromedia.com/labs/archives/2007/11/flash_player_9_6.cfm
• Team members actively testing betas
  • marcia
  • abillings
  • stephend

Additional References

New FF3 Feature list - An incomplete list of what Firefox 3 developers are working on.