canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,714
edits
Auto-tools/Automation Development/Meetings/130422: Difference between revisions
Auto-tools/Automation Development/Meetings/130422 (view source)
Revision as of 09:23, 23 April 2013
, 23 April 2013→Action items
Dpetrovici (talk | contribs) |
|||
| (24 intermediate revisions by 7 users not shown) | |||
| Line 15: | Line 15: | ||
= Highlights = | = Highlights = | ||
* [ | * [dave] Marionette client 0.5.24 released with several fixes for running tests on B2G desktop builds | ||
* New blockers, any updates? | * New blockers, any updates? | ||
** | ** {{bug|862724}} - Additional pulse message for same nightly builds sent (with previous_buildid == buildid) | ||
** {{bug|864294}} - Activities database is not automatically recreated if it has been removed | |||
* You can find all blockers here: http://bit.ly/Tph0NY | * You can find all blockers here: http://bit.ly/Tph0NY | ||
= Roundtable = | = Roundtable = | ||
* | * How to implement Metro tests, as it looks like we will majorly be blocked on missing event actions in Mozmill. Should we consider Marionette instead? | ||
* [henrik] Really awful proposal - Shall we drop work on Mozmill and start with Marionette for tests? | |||
** pro arguments | |||
*** Only a few tests are using complex mouse events like drag&drop, and we do not support touch events at all in Mozmill | |||
*** Marionette gets injected in the system process as extra library and doesn't have to be build as extension, which even is not supported for all products (e.g native web apps don't allow add-ons to be installed) | |||
*** W3C compliance by using the web driver protocol (http://www.w3.org/TR/2013/WD-webdriver-20130117/) | |||
*** Tests can even be run on mobile (Android, B2G) | |||
*** Tests can be written in different languages (e.g. Python) and allow us to write system integration tests because we are not in the browser scope anymore | |||
*** Support for (remote) debugging | |||
*** Changes of getting a much broader community around desktop testing | |||
*** Marionette has more active contributors than Mozmill | |||
** contra | |||
*** First we have to figure out how to inject Marionette in default Firefox builds (see {{bug|836253}}) | |||
*** Existing Mozmill tests would have to be transformed or we keep and run them in parallel (some tests would even be not necessary anymore due to new browser chrome tests) | |||
*** Where would we publish our test results? Our dashboard would need a rewrite or we can directly push to tbpl | |||
**** Probably TBPL2 (Tree Herder) | |||
* Mozmill 2.0 triage part 2 | |||
= Goals Overview = | = Goals Overview = | ||
| Line 31: | Line 48: | ||
* Goals progress: | * Goals progress: | ||
* TBD | * TBD | ||
* [dave] Marionette client 0.5.24 released with several fixes for running against B2G desktop builds | |||
* [dave] Database cleanup pull is currently blocked by {{bug|864294}} | |||
* [rwood] Submitted pull request for gaia-ui Bluetooth device discoverable test | |||
* [rwood] Gaia-ui endurance tests: | |||
** First results submissions to DataZilla b2g project | |||
** Setting up jenkins on my test PC | |||
** Tests submitted for review | |||
** Once the tests are in the gaia-ui repo and more results are on DataZilla, I will give a talk about them in our meeting, write a blog post, etc. | |||
== WebAPI (Rob) == | == WebAPI (Rob) == | ||
* | * No updates this week | ||
== WebRTC (Henrik) == | == WebRTC (Henrik) == | ||
* | * Caught a major regression ({{bug|863687}}) and leak for data channels last week (a single char change) | ||
* | * Framework 70% done, but needs further cleanup before the first review | ||
= Softvision Updates = | = Softvision Updates = | ||
== Desktop Automation == | == Desktop Automation == | ||
* Goals progress: | * Goals progress: Mozmill 2.0 release | ||
* Fixed: | * Fixed: | ||
** {{bug|708270}} - | ** {{bug|757827}} - Preferences are being lost between application restarts | ||
** {{bug|853478}} - Junit report generation fails if Mozmill is not able to send the report to the dashboard | |||
** {{bug|848577}} - Create a Mozmill Endurance test to catch regular expression memory leak (bug 837845) | |||
** {{bug|860330}} - Test failure "searchTerm.getNode(...) is null" in testAwesomeBar/testLocationBarSearches.js | |||
** {{bug|708270}} - Flash endurance tests expose potential performance degradation regression | |||
** {{bug|803489}} - Software update tests on Windows 8 fail sometimes due to updater prompt on startup (jsbridge cannot connect) | |||
** {{bug|826251}} - Make testPreferences/testPreferredLanguage.js language independend | |||
** {{bug|859746}} - Mutt javascript tests do not print the RESULTS block | |||
* Blocked bugs: | * Blocked bugs: | ||
| Line 53: | Line 84: | ||
* Working on: | * Working on: | ||
** [https://github.com/mozilla/mozmill-ci/issues/229 Mozmill-ci] - Log request/response bodies of update requests | |||
*** trying with EnvInject plugin | |||
** {{bug|860664}} - controller.waitThenClick() does not correctly wait for the element | |||
** {{bug|860655}} - elementslib.ID() fails to grab the element - "could not find element ID: id" | |||
** {{bug|732353}} - Disable all Discovery Pane tests due to unpredictable web dependencies | ** {{bug|732353}} - Disable all Discovery Pane tests due to unpredictable web dependencies | ||
** {{bug|857551}} - Create HTML page containing all the elements necessary for L10N testing of the context menu access keys | ** {{bug|857551}} - Create HTML page containing all the elements necessary for L10N testing of the context menu access keys | ||
** {{bug|671476}} - | ** {{bug|671476}} - Mozmill Endurance test for loading a SWF video via URL | ||
** {{bug|671479}} - | ** {{bug|671479}} - Mozmill Endurance test for loading a SWF video via iframe | ||
** {{bug|860382}} - Test failure "Search engines drop down has been closed" in testSearchSuggestions.js | ** {{bug|860382}} - Test failure "Search engines drop down has been closed" in testSearchSuggestions.js | ||
** {{bug|860659}} - | ** {{bug|860659}} - elementslib.Lookup() fails with 'Expression \"id(\"main-window\")\" returned null. Anonymous == false"' | ||
** {{bug|791634}} - | ** {{bug|791634}} - Make waitFor() method available in the assertions module (also for soft assertions) | ||
** {{bug|763461}} - Mozmill tests for cut-off elements should not report failure for scrollable direction | |||
** {{bug|763470}} - False positive test for cut-off elements for advanced pane tabs | |||
** {{bug|764640}} - Mozmill returns PASS/SKIP/FAIL == 0 for test_assertions.js | |||
** {{bug|860670}} - mozmill.utils.getWindowByType fails for type: Addons:Blocklist - Fails with "Window has been found." | |||
** {{bug|860662}} - controller.select() fails to select the option by value | |||
{| border="1" cellpadding="2" align="center" | {| border="1" cellpadding="2" align="center" | ||
| Line 74: | Line 114: | ||
|- | |- | ||
| {{bug|838192}} || Review || Henrik/Dave || Add screenshot functionality to assertion module (low priority with mozmill 2.0 looming) | | {{bug|838192}} || Review || Henrik/Dave || Add screenshot functionality to assertion module (low priority with mozmill 2.0 looming) | ||
|- | |- | ||
| {{bug|785014}} || Need-info || Alexandre Poirot || Build restartless.xpi as a plain bootstrapped extension | | {{bug|785014}} || Need-info || Alexandre Poirot || Build restartless.xpi as a plain bootstrapped extension | ||
|- | |- | ||
| {{bug|859746}} || Review || Henrik/Dave || Mutt javascript tests do not print the RESULTS block | | {{bug|859746}} || Review || Henrik/Dave || Mutt javascript tests do not print the RESULTS block | ||
| Line 92: | Line 124: | ||
|- | |- | ||
| {{bug|857551}} || Feedback || Henrik/Dave || Create HTML page containing all the elements necessary for L10N testing of the context menu access keys | | {{bug|857551}} || Feedback || Henrik/Dave || Create HTML page containing all the elements necessary for L10N testing of the context menu access keys | ||
|- | |||
| {{bug|860665}} || Feedback || Henrik/Dave || controller.waitForPageLoad() fails when working with Private Browsing windows. | |||
|- | |||
| {{bug|827276}} || Needinfo || Alex Keybl || Test failure 'Check for updates has been completed' in /restartTests/testSoftwareUpdateAutoProxy/test2.js | |||
|} | |} | ||
== Mobile (Robocop) == | == Mobile (Robocop) == | ||
* | * Progress on [https://docs.google.com/spreadsheet/ccc?key=0AhE7m4JB2j6tdDJBT2dlbVJwUk9PSy1RbHo4WVNiUGc#gid=8 Robocop Automation] | ||
* | * Bugs with work in progress: | ||
** {{Bug|820859}} - Robocop: Add test for 'Clear Private Data' | |||
** {{Bug|830834}} - Robocop: Add test for 'Show Search Suggestions' feature | |||
** {{Bug|846340}} - Robocop: Add test for 'Clear Site Settings' | |||
** {{Bug|836341}} - Robocop: Add test for "Clear download history" | |||
** {{Bug|838596}} - Robocop: Add test for 'Master Password' feature | |||
** {{Bug|846296}} - Robocop: Add test for 'Add Search Engine' feature | |||
** {{Bug|833832}} - Robocop: Add test for Import from Android feature | |||
** {{Bug|855978}} - Robocop: upgrade to robotium-solo-4.0.jar | |||
* Bugs awaiting review: | |||
**{{Bug|864280}} - Robocop: Cleanup of BaseTest.java.in | |||
* Intermittent fails we investigated: | |||
** {{Bug|836818}} - Intermittent testOverscroll | Checking gecko didn't draw unnecessarily - got true, expected false | |||
** {{Bug|851861}} - Intermittent testOverscroll, testPanCorrectness, testAxisLocking, testLoad, testFlingCorrectness | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0) | |||
** {{Bug|854043}} - Intermittent testWebContentContextMenu | Verify that the download started - The download started | |||
* New issues found during investigations | |||
** {{Bug|862695}} - Robocop testBrowserProvider test restarts at some point locally and has a lot of database access errors | |||
*Notes/Issues/Questions: | |||
** We still don't have an update on the robocop build issue - {{Bug|855306}} - Robocop build does not re-build tests - still no update from :joey - we are still working with older revisions or rebuilding everything | |||
** We are still seeing issues with the "Private data cleared" toaster notification not being displayed - {{Bug|837274}}. The "Clear private data" tests on tryserver seem to constantly fail. | |||
** We are still trying to reproduce intermittent issues and to find out why they are happening | |||
= Personal Status = | = Personal Status = | ||
| Line 107: | Line 163: | ||
==Action items== | ==Action items== | ||
* {{new|}} | * {{prev|}} Henrik - Update the goals page | ||
* {{prev|}} Anthony: Ensure any Metro tests are suitably tagged with in-testsuite? | |||
* {{new|}} Andreea: File bug about the broken elementslib handling with elements | |||