|
|
| Line 25: |
Line 25: |
| * Andreas Tolfsen (ato) - automation developer | | * Andreas Tolfsen (ato) - automation developer |
|
| |
|
| = Requirements = | | = Roadmap = |
|
| |
|
| == Marionette enhancements ==
| | * finish writing Marionette update and security tests [whimboo, chmanchester] |
| | | * get the above tests running in mozmill-CI and work on stabilizing them [whimboo] |
| # e10s support for Marionette, {{bug|1046512}}, [chmanchester]
| | ** rewrite mozmill automation scripts ( https://github.com/mozilla/mozmill-automation/ ), [whimboo] |
| # modal dialog support, {{bug|906712}}, [ato]
| | ** update the pre-configured environments ( https://github.com/whimboo/mozmill-environment ), [whimboo] |
| # restart capability, including passing est runtime data between processes, {{bug|940954}}, [AutomatedTester]
| | ** update mozmill-ci to handle marionette instead of mozmill, [whimboo] |
| # anonymous node support in chrome, {{bug|1080764}}, [ahal]
| | ** revise documentation, [whimboo] |
| # be able to use l10n lookups; needs further investigation, {{bug|1088116}}, [chmanchester]
| | * uplift recent Marionette changes to aurora so that we won't have to depend on mozmill for release testing of the nest ESR [AutomatedTester] |
| # be able to install and work with addons (may need minor runner changes), [owner TBD]
| | * implement method of running Marionette update tests that span different gecko releases [chmanchester] |
| # be able to report results to mozmill dashboard (add a runner mixin?), [owner TBD]
| | * work with Releng to get Marionette update tests running in buildbot release automation [owner: TBD] |
| | | ** in parallel, work with Treeherder to enable it to display jobs from buildbot release automation |
| == Mozmill infrastructure and CI changes ==
| | * work with Releng to get security and other Marionette tests running in buildbot release automation |
| | | * update the mozmill-crowd add-on to let community still participate in testing (probably low priority), [whimboo] |
| # rewrite mozmill automation scripts ( https://github.com/mozilla/mozmill-automation/ ), [whimboo]
| | * rework Mozmill tests for TPS to become Marionette tests (low priority), [whimboo] |
| # update the pre-configured environments ( https://github.com/whimboo/mozmill-environment ), [whimboo]
| | * gradually turn off tests in mozmill-CI that are covered by m21s tests running in buildbot, with the goal of being completely off of mozmill-CI within two quarters [whimboo] |
| # update mozmill-ci to handle marionette instead of mozmill, [whimboo]
| |
| # update the mozmill-crowd add-on to let community still participate in testing, [whimboo]
| |
| # revise documentation, [whimboo]
| |
| # rework Mozmill tests for TPS to become Marionette tests (probably low priority), [whimboo]
| |
|
| |
|
| == Test conversion == | | == Test conversion == |
| We will convert a small number of tests that don't rely on outstanding marionette enhancements, as a POC. The plan for conversion of the remaining tests is still TBD. | | We are prioritizing the conversion of update and security tests; others will follow later as time and resources permit. |
|
| |
|
| The repository this work is happening can be found at https://github.com/mozilla/firefox-ui-tests. | | The repository this work is happening can be found at https://github.com/mozilla/firefox-ui-tests. |
Introduction
m21s is the code name of the mozmill-marionette-e10s project, the project whose aim is to replace mozmill with e10s-compliant marionette tests. The project consists roughly of the following pieces:
- adapt marionette to e10s
- add features to marionette required by mozmill tests
- rewrite existing mozmill tests as marionette tests
- update the mozmill infrastructure and CI to utilize the new tests and harness
Many of these pieces can be done in parallel, but this is a significant project which will likely take multiple months to fully deliver.
Rationale
Mozmill is currently used for release and update testing by QA. It provides coverage not duplicated elsewhere, and sometimes finds important bugs. Mozmill was written long before e10s and features a model which includes promiscuous chrome-content interactions implemented in a way which is not trivial to port to an e10s-compatible model. Making mozmill e10s-compatible would involve updating the harness, the libraries and the test themselves.
Although this is achievable, there has been momentum building for the idea of migrating mozmill on top of marionette, to benefit from the stability of marionette's protocol. An evaluation of this idea has been performed, and it was determined that this solution would be difficult to implement and likely problematic as well. Combined with the requirement of e10s-compatibility, this idea was determined to be not worth the effort and has been abandoned in favor of simply rewriting all of the existing mozmill tests as marionette tests, after the necessary feature work to marionette has been accomplished.
Team
- Jonathan Griffin (jgriffin) - project management
- Henrik Skupin (whimboo) - mozmill maintainer and infrastructure/CI owner
- David Burns (AutomatedTester) - Marionette lead
- Andrew Halberstadt (ahal) - automation developer
- Chris Manchester (chmanchester) - automation developer
- Andreas Tolfsen (ato) - automation developer
Roadmap
- finish writing Marionette update and security tests [whimboo, chmanchester]
- get the above tests running in mozmill-CI and work on stabilizing them [whimboo]
- uplift recent Marionette changes to aurora so that we won't have to depend on mozmill for release testing of the nest ESR [AutomatedTester]
- implement method of running Marionette update tests that span different gecko releases [chmanchester]
- work with Releng to get Marionette update tests running in buildbot release automation [owner: TBD]
- in parallel, work with Treeherder to enable it to display jobs from buildbot release automation
- work with Releng to get security and other Marionette tests running in buildbot release automation
- update the mozmill-crowd add-on to let community still participate in testing (probably low priority), [whimboo]
- rework Mozmill tests for TPS to become Marionette tests (low priority), [whimboo]
- gradually turn off tests in mozmill-CI that are covered by m21s tests running in buildbot, with the goal of being completely off of mozmill-CI within two quarters [whimboo]
Test conversion
We are prioritizing the conversion of update and security tests; others will follow later as time and resources permit.
The repository this work is happening can be found at https://github.com/mozilla/firefox-ui-tests.
Bugs
No results.
0 Total;
0 Open (0%);
0 Resolved (0%);
0 Verified (0%);
Full Query
| ID |
Priority |
Summary |
Assigned to |
| 1107336 |
P1 |
Split marionette automation API into its own package and not have a monolithic package |
David Burns :automatedtester |
| 906712 |
P1 |
implement tab modal dialog handling for desktop with Marionette (alert, confirm, prompt) |
Chris Manchester (limited bugmail, email directly) |
| 940954 |
P1 |
Let Marionette support shutdown/restart tests |
David Burns :automatedtester |
| 941749 |
P1 |
getWindows should return tabs |
Chris Manchester (limited bugmail, email directly) |
| 1169180 |
P1 |
Intermittent failure for update tests and unit tests in test_about_window.py due to Marionette exception: "win.document.documentElement is null" |
Chris Manchester (limited bugmail, email directly) |
| 1129702 |
P1 |
Add support for doubleclicking to marionette's actions |
Chris Manchester (limited bugmail, email directly) |
| 1106913 |
P1 |
Detect cyclic object references when marshaling objects in evaluate.toJSON |
Andreas Tolfsen ❲:ato❳ |
| 1103196 |
P1 |
Support for untrusted/self-signed TLS certificates |
Andreas Tolfsen ❲:ato❳ |
| 1097705 |
P1 |
Add ability to right or middle click an element |
Chris Manchester (limited bugmail, email directly) |
| 1046512 |
P3 |
Marionette breaks in e10s enabled Nightly |
Chris Manchester (limited bugmail, email directly) |
| 1088116 |
-- |
Marionette support for testing l10n builds at parity with mozmill |
Chris Manchester (limited bugmail, email directly) |
| 1091662 |
-- |
Ensure shortcuts work in Marionette tests |
Chris Manchester (limited bugmail, email directly) |
| 1092223 |
-- |
Marionette tests fail with e10s enabled |
|
| 1093707 |
-- |
Using navigate() in chrome scope locks up the browser |
Chris Manchester (limited bugmail, email directly) |
| 1094441 |
-- |
Enable 'css selector' strategy in chrome scope |
David Burns :automatedtester |
| 1096488 |
-- |
Marionette support for navigating between remote and non-remote pages with e10s enabled |
Chris Manchester (limited bugmail, email directly) |
| 1096571 |
-- |
Don't call into Webdriver's isShown() method when attempting to manipulate content xul elements |
Andrew Halberstadt [:ahal] |
| 1084412 |
-- |
searchTimeout doesn't work when using findElements |
Andrew Halberstadt [:ahal] |
| 1099331 |
-- |
Marionette's navigate errors on loading an about:blocked or about:error page |
Chris Manchester (limited bugmail, email directly) |
| 1083131 |
-- |
Marionette does not always remove temporary profiles |
Chris Manchester (limited bugmail, email directly) |
| 1104742 |
-- |
Stopping Marionette with Ctrl-C does not quit the application |
Andrew Halberstadt [:ahal] |
| 1080772 |
-- |
Convert a few mozmill tests to Marionette as POC |
Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08) |
| 755036 |
-- |
Save globals between execute_script calls in 'chrome' contexts |
Chris Manchester (limited bugmail, email directly) |
| 1114623 |
-- |
Implement closeChromeWindow() to close the currently active chrome window |
Chris Manchester (limited bugmail, email directly) |
| 1120630 |
-- |
Turn on Mn-e10s |
Chris Manchester (limited bugmail, email directly) |
| 1120809 |
-- |
Marionette sometimes fails to register a content script with e10s enabled |
Chris Manchester (limited bugmail, email directly) |
| 1122609 |
-- |
execute_script() called with new_sandbox=False in chrome context does not use current script_args but those from a former call |
Chris Manchester (limited bugmail, email directly) |
| 1123683 |
-- |
Set default testing preferences (e.g. disabling FHR and other metric pings) |
Barbara Miller (:galgeek) |
| 1124604 |
-- |
Add `focus` parameter to switch_to_window() to stay in compat with switch_to_frame() |
Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08) |
| 1124667 |
-- |
Release marionette-driver 0.2 and marionette-client 0.9 |
Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08) |
| 1125209 |
-- |
Sending a string with capital letters should not trigger a shift key |
|
| 1125377 |
-- |
Marionette fails during getWindowHandles with e10s turned on when accessing a CPOW |
Chris Manchester (limited bugmail, email directly) |
| 1080764 |
-- |
Add support for anonymous nodes in chrome |
Andrew Halberstadt [:ahal] |
| 1135846 |
-- |
Re-organize marionette actions code to expose actions to chrome scope |
Chris Manchester (limited bugmail, email directly) |
| 1137388 |
-- |
Marionette hard-kills the browser for restarts which causes lots of trouble |
Chris Manchester (limited bugmail, email directly) |
| 1141483 |
-- |
After a restart Marionette doesn't restore the previous context |
Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08) |
| 1141519 |
-- |
Marionette hangs if exceptions are happening in content scope |
David Burns :automatedtester |
| 1141679 |
-- |
Marionette doesn't clean up its socket following a restart initiated inside the application |
Chris Manchester (limited bugmail, email directly) |
| 1144907 |
-- |
Marionette's gather_debug shouldn't leave calling code in content scope if it didn't start there |
Chris Manchester (limited bugmail, email directly) |
| 1028407 |
-- |
Try to make marionette connect to browser sessions quicker |
Chris Manchester (limited bugmail, email directly) |
| 1169600 |
-- |
"MarionetteException: this.browserForTab is null" when closing the first tab of a chrome window |
Chris Manchester (limited bugmail, email directly) |
| 1169798 |
-- |
MarionetteException in switch_to_window() with chrome scope enabled: "this.browser is undefined" |
Chris Manchester (limited bugmail, email directly) |
| 1176663 |
-- |
[e10s] Marionette.navigate() hangs when first tab gets closed with: NS_ERROR_ILLEGAL_VALUE in nsIMessageSender.sendAsyncMessage |
|
| 1176758 |
-- |
Allow mozprocess to detect and kill detached child processes |
Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08) |
| 1202392 |
-- |
Improve exception handling and message details in Marionette |
Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08) |
45 Total;
45 Open (100%);
0 Resolved (0%);
0 Verified (0%);