QA/MergeCompatibilityTesting

Merge Compatibility Testing

Introduction

The fast release cycle has proven to be challenging to validate platform and web compatibility from release to release. Regressions are more likely to occur undetected than in the prior longer release model prior to shipping Firefox 5. To get a better assessment of Firefox compatibility, QA in conjunction with our testing community and with outsourced testing partners have been building up experience and expertise in conducting both web and platform compatibility test runs.

Objective

The objecting of the Merge Compatibility Project is to perform a comprehensive set of regression tests for both web and platform compatibility across the current Firefox release channels. Web compatibility refers to crowd-sourced manual tests of the most visited global web sites and perform user based scenario interactions. The selected web sites are grouped by functionality. For instance if it is a financial site, the objective would be for a tester to log into to the site a perform an actual transaction and verify the results of the transaction. If it is a music site, the tester should select and play an audio file on the site and so on. The objective is detect deeper issues that may arise for a realistic user interaction with the sites features, pages and capabilities.

The objective of platform compatibility is to ensure Firefox behaves correct across the spectrum of supported hardware and platform configurations. Historically we have relied on our test community to provide the coverage of testing across the matrix of hardware platforms and supported OSes. This reliance has proven to lack completeness and therefore requires a different approach. The proposed approach is to use an outsource vendor (Ibeta) to supply the machines and configurations which to run graphics and targeted OS compatibility tests.

Implementation

The merge compatibility testing will be conducted on each release channel (nightly, aurora and beta) 3-4 weeks following a channel software merge event. The compatibility testing is broken down into two distinct focus areas. The first area is web compatibility testing using a crowd-sourced approach.

Web Compatibility Testing

Desktop Firefox Web Compatibility 2011-08-16

• results spreadsheet
• reported bugs

Desktop Firefox Web Compatibility 2011-10-25

• results spreadsheet

Platform Compatibility Testing

12-02-2011 Graphics Compatibility and Blocklist Testing

Blocklist testing has been requested by Doug Sherk who did some change and code cleanup of the graphics blocklist feature.

From doug's email: 11-23-11
I've made some changes to driver blocklisting recently which hit every platform and enable a downloadable blocklist on them. There are tons of edge cases here which I don't have the hardware to test and I think this would benefit from a proper QA run; it would be pretty bad if there were mistakes here, since users could be blocked from features even if their drivers are fine, and others could be allowed to use features that are supposed to be blocked. This whole effort is done primarily to allow blocklisting Android devices, but as a side effect, other OS's can now use this system.

• For reference here are the bugs relevant to this:
  • 668004- Generalize the downloaded GPU blacklist to non-Windows platforms
  • 699482 - Talos complains about GfxDriverInfo static constructors, they might slow down startup
  • 704710 - Refactor GfxDriverInfo/GfxInfo(Base) to support string device and vendor IDs
  • 689598 - Implement downloadable device blacklisting for Android
  • 700124 - Attempting to use JNI in child process [@ mozilla::AndroidBridge::EnsureJNIThread

In Addition to the blocklist testing, we'll conduct a graphics compatibility test with our outsource vendor IBeta to conduct a comprehensive graphics and system compatibility across the current channel Firefox versions:

• Nightly - FF 11
• Aurora - FF 10
• Beta - FF 9

The machine configurations that will be used in this testing are listed in the google spreadsheet

Results

Links and Resources

• on going meeting etherpad
• https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers
• https://wiki.mozilla.org/Blocklisting/Testing
• Machine Configuration Matrix
• Merge Compatibility Test Definitions