Auto-tools/Projects/Halreftest
Introduction
Halreftest (for Hardware Acceleration & Layers Reftest) is an extension being developed to enable crowdsource testing of hardware acceleration and layers on a greater variety of hardware combinations that Mozilla could reasonably expect to have access to internally. The tool will initially be promoted during QA testdays at periodic intervals, and may be promoted for wider distribution via some other channel after the graphics code has reached a desired level of stability. (what channel, and what criteria will we use to determine if it's ready for wider distribution?)
When a user invokes the Halreftest extension, it displays a welcome page which describes the purpose of the extension and provides a "Start" button. When the user clicks this, the extension will run a series of reftests which are bundled into the extension. When the tests are complete, the extension will post the test results, along with some system data, to a database at Mozilla, where it can later be analyzed.
What It Tests
As its name implies, Halreftest is designed primarily to run reftests. According to developers on the graphics team, it should be possible using this method to achieve reasonable coverage of the relevant code. Currently Halreftest runs a subset of SVG tests, but this can be easily changed. Although the current design runs reftests in the standard way, the plan is to run them as follows:
- hardware accelerated reftest test compared to accelerated reftest reference
- hardware accelerated reftest test compared to non-accelerated reftest test
This will be implemented as soon as we can toggle hardware acceleration and layers support in the test harness.
For #2 above, if a test fails, it may be due to small differences in the rendering engines used, which are not visible to the human eye. We really only care about rendering differences which are visible and obvious. To handle this case, failures of this type will be presented to the tester, with both images side-by-side, and the tester will be asked to determine if the images are visually identical or not. If they are, the test will be marked as passed.
All of the above tests are only designed to test rendering correctness. It is also desirable to test rendering performance, but there currently exists no easy way to do this. Graphics team developers have stated they may implement some type of system that could profile time spent in Cairo code, and if this is done, it could easily be added to Halreftest in order to provide performance metrics.
Collecting Test Data
All the data generated by Halreftest will be posted to a MySQL database at the end of the test. This db is currently up and running, although no interesting views have been created yet; viewing data is still a matter of executing SQL queries. We plan on creating database views that allow developers to easily correlate failures with specific hardware features.
The data collected includes all tests run and their results, and in case of test failure, the test images. We also collect various system stats to post along with the test results:
- OS
- Video card brand and name
- Video chipset and dactype
- Video RAM
- Video driver: files, date, and version
- Video display settings
The above have been implemented for Win32, and porting to Mac is currently in progress. Porting to linux is also planned. Some system stats may vary depending on OS (e.g., it appears that the video driver on OSX is an intrinsic part of the OS, and thus doesn't have a distinct date and version).
We also plan to allow the tester to optionally include their e-mail address (in case we have followup questions regarding results on their particular hardare), and comments (to allow testers to note problems not directly related to individual tests). See bug 548884.
The User Interface
The current UI borrows graphic elements from other Mozilla properties. We've asked Sean Martell if he could possibly create an interesting icon and better name for the extension, but suggestions from other sources are welcome too!
The welcome page
The system information panel
The comparison screen
