Confirmed users
1,927
edits
Auto-tools/Projects/SpeedTests: Difference between revisions
Jump to navigation
Jump to search
no edit summary
(Created page with "== Description == SpeedTests is an automated suite of browser tests in which all tests are executed on a series of browsers. Tests can be freeform but are expected to a) report...") |
No edit summary |
||
| Line 1: | Line 1: | ||
== Description == | == Description == | ||
SpeedTests is an automated suite of browser tests in which all tests are executed on a series of browsers. Tests can be freeform but are expected to a) report results and b) have a definite termination point, for which purpose a small set of JavaScript functions are provided. Likely candidates are the IE 9 Test Drive performance tests. | SpeedTests is an automated suite of browser tests in which all tests are executed on a series of browsers. Tests can be freeform but are expected to a) report results and b) have a definite termination point, for which purpose a small set of JavaScript functions are provided. Likely candidates are the IE 9 Test Drive performance tests, suitably modified. | ||
== Architecture == | == Architecture == | ||
| Line 10: | Line 9: | ||
The browser controller has a list of browsers likely to be installed on the target platform. It iterates through the list, executing the browser if it is found, pointing it to the server path /nexttest/, which replies with a redirect to the first test (in string order). It also has a simple HTTP server running on a special port, which awaits a callback from the browser when the last test has executed. | The browser controller has a list of browsers likely to be installed on the target platform. It iterates through the list, executing the browser if it is found, pointing it to the server path /nexttest/, which replies with a redirect to the first test (in string order). It also has a simple HTTP server running on a special port, which awaits a callback from the browser when the last test has executed. | ||
The tests use a common set of JavaScript functions to report results and to load the next test. | The tests use a common set of JavaScript functions to report results and to load the next test. The server collects results and serves tests. The client sends the results via XHRs to the server, either once at the end of the test, or periodically throughout the test's duration (TODO: possibly store periodic results internally and send once instead of multiple times, if it affects test execution). When the test is finished, the browser loads the URL /nexttest/<current test name>, eg. /nexttest/PsychedelicBrowsing/, which redirects to the test that, in string order, follows the current test. If the current test was the last test, the server redirects the browser to the callback port on the localhost. | ||
Upon receiving a GET request on the callback port, the browser controller terminates the current browser and launches the next one in the list. When the last browser has executed the tests, the controller shuts down its HTTP server and exits. | Upon receiving a GET request on the callback port, the browser controller terminates the current browser and launches the next one in the list. When the last browser has executed the tests, the controller shuts down its HTTP server and exits. | ||
== Implementation == | == Implementation == | ||
| Line 22: | Line 20: | ||
Tests are HTML, JavaScript, and associated files. The server also provides a file containing several JavaScript functions to be used to report results and load subsequent tests. | Tests are HTML, JavaScript, and associated files. The server also provides a file containing several JavaScript functions to be used to report results and load subsequent tests. | ||
== Issues == | == Issues == | ||
| Line 28: | Line 25: | ||
As with any automation, there are finnicky things needing to be worked out (or around): | As with any automation, there are finnicky things needing to be worked out (or around): | ||
* Window size. For a proper comparison, we want all browsers to run in the same-sized window. However, not all OSs appear to provide a way to dictate window size when launching an application ( | * Window size. For a proper comparison, we want all browsers to run in the same-sized window. However, not all OSs appear to provide a way to dictate window size when launching an application (testing so far has been on OS X, for which I have not been able to find a suitable solution). | ||
* Security preferences. Some browsers, notably Opera, don't like pages that try to redirect to localhost. Some prompt the user to restore the previous session upon launch. This requires some sort of initial setup; hopefully a saved profile will be sufficient. Although, Opera sometimes forgets security settings and reverts to user prompts--not useful in an automated environment. | * Security preferences. Some browsers, notably Opera, don't like pages that try to redirect to localhost. Some prompt the user to restore the previous session upon launch. This requires some sort of initial setup; hopefully a saved profile will be sufficient. Although, Opera sometimes forgets security settings and reverts to user prompts--not useful in an automated environment. | ||
* Frozen tests. It's entirely possible that a test won't finish, either because of a problem in the test or wonky browser behaviour. The controller could kill the browser in this case, but the controller doesn't know how long the current test has been running (or even which test it's executing). Furthermore, we'd probably lose the rest of the test results.<br><br>A possible solution would be to move the test-loading logic to the controller instead of being done in the test itself. However this prevents a casual user from running the test suite independent of a controller (by just going to http://.../nexttest/). It also requires finer control over the browser. | * Frozen tests. It's entirely possible that a test won't finish, either because of a problem in the test or wonky browser behaviour. The controller could kill the browser in this case, but the controller doesn't know how long the current test has been running (or even which test it's executing). Furthermore, we'd probably lose the rest of the test results.<br><br>A possible solution would be to move the test-loading logic to the controller instead of being done in the test itself. However this prevents a casual user from running the test suite independent of a controller (by just going to http://.../nexttest/). It also requires finer control over the browser. | ||