Compatibility/Automated compat testing: Difference between revisions
Jump to navigation
Jump to search
m (Roadmap clarification) |
|||
| Line 15: | Line 15: | ||
# Improve client scripts - implement scaling up to n tabs functionality | # Improve client scripts - implement scaling up to n tabs functionality | ||
# Automated tests for the framework | # Automated tests for the framework | ||
# Support getting | # Support getting URLs directly from Bugzilla or Webcompat searches for a test run (likely in a client script) | ||
# Support AWCY list mode - running all sites on an [http://arewecompatibleyet.com AWCY] list (likely in client script) | # Support AWCY list mode - running all sites on an [http://arewecompatibleyet.com AWCY] list (likely in client script) | ||
# Do live CSS analysis across Gecko+WebKit, find a suitable way to store this information | # Do live CSS analysis across Gecko+WebKit, find a suitable way to store this information | ||
Revision as of 20:24, 4 August 2015
About
The tool Compatipede (built on SlimerJS for Gecko and PhantomJS for Webkit rendering) can do automated testing of live websites.
The framework consists of the projects jannah for the infrastructure that's capable of scaling up to testing many pages at once across many computers and boar for the script that controls SlimerJS/PhantomJS.
Roadmap
Compatipede 2 is still under development. This is a prioritised list of tasks we would like to complete:
- Implement support for plugins [1] - specific snippets of JS to check for specific problems
- Database for result storage (compat-entomology re-use or similar schema)
- Integrate with DB for result storage
- Generating reports from DB contents
- Improve client scripts - implement scaling up to n tabs functionality
- Automated tests for the framework
- Support getting URLs directly from Bugzilla or Webcompat searches for a test run (likely in a client script)
- Support AWCY list mode - running all sites on an AWCY list (likely in client script)
- Do live CSS analysis across Gecko+WebKit, find a suitable way to store this information
- Track and report JS errors (preferably flagging those that happen in one of the engines but not the other)
- Track and report redirects
- Way to run Hallvord's existing regression tests from the old extension / slimertester.js
- Selenium / WebDriver support to facilitate easier automation, e.g. log in
Re-purposing Compatipede 2
The above features will be excellent for web compatibility testing. However, the tool we write can also be used in various other ways, including
- Stability testing - loading many sites w/o crashing
- Browser regression testing - comparing output from Gecko version n and Gecko version n+1
- Memory analysis - using Gecko's built-in methods for dumping memory stats or OS reporting
- A special case of memory analysis might be to compare memory usage for Gecko and Webkit when loading the same page, might give us hints about structures and sites Webkit handles better than Gecko memory-wise
Compatipede 2 architecture
