Auto-tools/Projects/Halreftest: Difference between revisions
| (12 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
== Introduction == | == Introduction == | ||
Halreftest | Grafx Bot is the new Halreftest! After meeting with members of the design team, Grafx Bot was selected as the new name for this project. Sean Martell is going to produce a robot icon/logo for it as well. [https://bugzilla.mozilla.org/show_bug.cgi?id=549416 bug 549416]. | ||
When a user invokes the | Grafx Bot is an extension being developed to enable crowdsource testing of [[Platform/GFX/HardwareAcceleration|hardware acceleration]] and [[Gecko:Layers|layers]] on a greater variety of hardware combinations than Mozilla could reasonably expect to have access to internally. The tool will initially be promoted during QA testdays at periodic intervals, and may be promoted for wider distribution via some other channel after the graphics code has reached a desired level of stability. (what channel, and what criteria will we use to determine if it's ready for wider distribution?) | ||
When a user invokes the Grafx Bot extension, it displays a welcome page which describes the purpose of the extension and provides a "Start" button. When the user clicks this, the extension will run a series of reftests which are bundled into the extension. When the tests are complete, the extension will post the test results, along with some system data, to a database at Mozilla, where it can later be analyzed. | |||
<br> | |||
== Current Status [Mar 30, 2010] == | |||
* Extension code is now at http://bitbucket.org/jonallengriffin/grafxbot/ (mw22 update: this link is obsolete now. The code is now maintained at: http://hg.mozilla.org/automation/grafxbot ) | |||
* XPCOM component has been ported to Win, Mac & Linux | |||
* Simple database views now available at http://brasstacks.mozilla.com/halreftest/data/ | |||
* Tool is being readied for internal testing within the Tools & QA teams | |||
* Waiting on new icon/logo per [https://bugzilla.mozilla.org/show_bug.cgi?id=549416 bug 549416] | |||
* Waiting on privacy policy per [https://bugzilla.mozilla.org/show_bug.cgi?id=548884 bug 548884] | |||
* Need input from graphics team about which reftests we should be running | |||
* There's an assertion which occurs when starting the test on Windows; filed as [https://bugzilla.mozilla.org/show_bug.cgi?id=556119 bug 556119] | |||
<br> | <br> | ||
| Line 9: | Line 24: | ||
== What It Tests == | == What It Tests == | ||
Grafx Bot is designed primarily to run reftests. According to developers on the graphics team, it should be possible using this method to achieve reasonable coverage of the relevant code. Currently Grafx Bot runs a subset of SVG tests, but this can be easily changed. Normally the current design runs reftests in the standard way, but will run them as follows on platforms that support hardware acceleration (currently Windows only): | |||
#hardware accelerated reftest test compared to accelerated reftest reference | #hardware accelerated reftest test compared to accelerated reftest reference | ||
#hardware accelerated reftest test compared to non-accelerated reftest test | #hardware accelerated reftest test compared to non-accelerated reftest test | ||
For #2 above, if a test fails, it may be due to small differences in the rendering engines used, which are not visible to the human eye. We really only care about rendering differences which are visible and obvious. To handle this case, failures of this type will be presented to the tester, with both images side-by-side, and the tester will be asked to determine if the images are visually identical or not. If they are, the test will be marked as passed. | For #2 above, if a test fails, it may be due to small differences in the rendering engines used, which are not visible to the human eye. We really only care about rendering differences which are visible and obvious. To handle this case, failures of this type will be presented to the tester, with both images side-by-side, and the tester will be asked to determine if the images are visually identical or not. If they are, the test will be marked as passed. | ||
All of the above tests are only designed to test rendering correctness. It is also desirable to test rendering performance, but there currently exists no easy way to do this. Graphics team developers have stated they may implement some type of system that could profile time spent in Cairo code, and if this is done, it could easily be added to | All of the above tests are only designed to test rendering correctness. It is also desirable to test rendering performance, but there currently exists no easy way to do this. Graphics team developers have stated they may implement some type of system that could profile time spent in Cairo code, and if this is done, it could easily be added to Grafx Bot in order to provide performance metrics. | ||
<br> | <br> | ||
| Line 24: | Line 37: | ||
== Collecting Test Data == | == Collecting Test Data == | ||
All the data generated by | All the data generated by Grafx Bot will be posted to a MySQL database at the end of the test. There are some simple views available at http://brasstacks.mozilla.com/halreftest/data/, and more can be added as needed. | ||
The data collected includes all tests run and their results, and in case of test failure, the test images. We also collect various system stats to post along with the test results: | The data collected includes all tests run and their results, and in case of test failure, the test images. We also collect various system stats to post along with the test results: | ||
*OS | *OS | ||
*Video card brand | *Video card brand, name, device id, and vendor id | ||
*Video RAM | *Video RAM | ||
*Video driver: files, date, and version | *Video driver: files, date, and version | ||
*Video display settings | *Video display settings | ||
*Firefox build date and version | |||
Some system stats may vary depending on OS (e.g., it appears that the video driver on OSX is an intrinsic part of the OS, and thus doesn't have a distinct date and version). | |||
We also allow testers to optionally provide comments at the end of the test, relating either to Grafx Bot itself or the hardware acceleration and layer features. | |||
= The User Interface = | = The User Interface = | ||
The current UI borrows graphic elements from other Mozilla properties. | The current UI borrows graphic elements from other Mozilla properties. Suggestions for improvements are certainly welcome! | ||
<br> | <br> | ||
| Line 62: | Line 73: | ||
[[Image:Halreftest-comparison.png]] | [[Image:Halreftest-comparison.png]] | ||
<br> | |||
'''The 'Tests Complete' screen''' | |||
[[Image:Grafxbot-complete.png]] | |||
Latest revision as of 20:33, 18 April 2011
Introduction
Grafx Bot is the new Halreftest! After meeting with members of the design team, Grafx Bot was selected as the new name for this project. Sean Martell is going to produce a robot icon/logo for it as well. bug 549416.
Grafx Bot is an extension being developed to enable crowdsource testing of hardware acceleration and layers on a greater variety of hardware combinations than Mozilla could reasonably expect to have access to internally. The tool will initially be promoted during QA testdays at periodic intervals, and may be promoted for wider distribution via some other channel after the graphics code has reached a desired level of stability. (what channel, and what criteria will we use to determine if it's ready for wider distribution?)
When a user invokes the Grafx Bot extension, it displays a welcome page which describes the purpose of the extension and provides a "Start" button. When the user clicks this, the extension will run a series of reftests which are bundled into the extension. When the tests are complete, the extension will post the test results, along with some system data, to a database at Mozilla, where it can later be analyzed.
Current Status [Mar 30, 2010]
- Extension code is now at http://bitbucket.org/jonallengriffin/grafxbot/ (mw22 update: this link is obsolete now. The code is now maintained at: http://hg.mozilla.org/automation/grafxbot )
- XPCOM component has been ported to Win, Mac & Linux
- Simple database views now available at http://brasstacks.mozilla.com/halreftest/data/
- Tool is being readied for internal testing within the Tools & QA teams
- Waiting on new icon/logo per bug 549416
- Waiting on privacy policy per bug 548884
- Need input from graphics team about which reftests we should be running
- There's an assertion which occurs when starting the test on Windows; filed as bug 556119
What It Tests
Grafx Bot is designed primarily to run reftests. According to developers on the graphics team, it should be possible using this method to achieve reasonable coverage of the relevant code. Currently Grafx Bot runs a subset of SVG tests, but this can be easily changed. Normally the current design runs reftests in the standard way, but will run them as follows on platforms that support hardware acceleration (currently Windows only):
- hardware accelerated reftest test compared to accelerated reftest reference
- hardware accelerated reftest test compared to non-accelerated reftest test
For #2 above, if a test fails, it may be due to small differences in the rendering engines used, which are not visible to the human eye. We really only care about rendering differences which are visible and obvious. To handle this case, failures of this type will be presented to the tester, with both images side-by-side, and the tester will be asked to determine if the images are visually identical or not. If they are, the test will be marked as passed.
All of the above tests are only designed to test rendering correctness. It is also desirable to test rendering performance, but there currently exists no easy way to do this. Graphics team developers have stated they may implement some type of system that could profile time spent in Cairo code, and if this is done, it could easily be added to Grafx Bot in order to provide performance metrics.
Collecting Test Data
All the data generated by Grafx Bot will be posted to a MySQL database at the end of the test. There are some simple views available at http://brasstacks.mozilla.com/halreftest/data/, and more can be added as needed.
The data collected includes all tests run and their results, and in case of test failure, the test images. We also collect various system stats to post along with the test results:
- OS
- Video card brand, name, device id, and vendor id
- Video RAM
- Video driver: files, date, and version
- Video display settings
- Firefox build date and version
Some system stats may vary depending on OS (e.g., it appears that the video driver on OSX is an intrinsic part of the OS, and thus doesn't have a distinct date and version).
We also allow testers to optionally provide comments at the end of the test, relating either to Grafx Bot itself or the hardware acceleration and layer features.
The User Interface
The current UI borrows graphic elements from other Mozilla properties. Suggestions for improvements are certainly welcome!
The welcome page
The system information panel
The comparison screen
The 'Tests Complete' screen
