Auto-tools/Projects/m21s

Introduction

m21s is the code name of the mozmill-marionette-e10s project, the project whose aim is to replace mozmill with e10s-compliant marionette tests. The project consists roughly of the following pieces:

• adapt marionette to e10s
• add features to marionette required by mozmill tests
• rewrite existing mozmill tests as marionette tests
• update the mozmill infrastructure and CI to utilize the new tests and harness

Many of these pieces can be done in parallel, but this is a significant project which will likely take multiple months to fully deliver.

Rationale

Mozmill is currently used for release and update testing by QA. It provides coverage not duplicated elsewhere, and sometimes finds important bugs. Mozmill was written long before e10s and features a model which includes promiscuous chrome-content interactions implemented in a way which is not trivial to port to an e10s-compatible model. Making mozmill e10s-compatible would involve updating the harness, the libraries and the test themselves.

Although this is achievable, there has been momentum building for the idea of migrating mozmill on top of marionette, to benefit from the stability of marionette's protocol. An evaluation of this idea has been performed, and it was determined that this solution would be difficult to implement and likely problematic as well. Combined with the requirement of e10s-compatibility, this idea was determined to be not worth the effort and has been abandoned in favor of simply rewriting all of the existing mozmill tests as marionette tests, after the necessary feature work to marionette has been accomplished.

Team

• Jonathan Griffin (jgriffin) - project management
• Henrik Skupin (whimboo) - mozmill maintainer and infrastructure/CI owner
• David Burns (AutomatedTester) - Marionette lead
• Andrew Halberstadt (ahal) - automation developer
• Chris Manchester (chmanchester) - automation developer
• Andreas Tolfsen (ato) - automation developer

Requirements

Marionette enhancements

1. e10s support for Marionette, bug 1046512, [chmanchester]
2. modal dialog support, bug 906712, [ato]
3. restart capability, including passing est runtime data between processes, bug 940954, [AutomatedTester]
4. anonymous node support in chrome, bug 1080764, [ahal]
5. be able to use l10n lookups; needs further investigation, bug 1088116, [chmanchester]
6. be able to install and work with addons (may need minor runner changes), [owner TBD]
7. be able to report results to mozmill dashboard (add a runner mixin?), [owner TBD]

Mozmill infrastructure and CI changes

1. rewrite mozmill automation scripts ( https://github.com/mozilla/mozmill-automation/ ), [whimboo]
2. update the pre-configured environments ( https://github.com/whimboo/mozmill-environment ), [whimboo]
3. update mozmill-ci to handle marionette instead of mozmill, [whimboo]
4. update the mozmill-crowd add-on to let community still participate in testing, [whimboo]
5. revise documentation, [whimboo]
6. rework Mozmill tests for TPS to become Marionette tests (probably low priority), [whimboo]

Test conversion

We will convert a small number of tests that don't rely on outstanding marionette enhancements, as a POC.

The plan for conversion of the remaining tests is still TBD.

Bugs

Open bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Fixed bugs

Full Query

ID	Priority	Summary	Assigned to
1107336	P1	Split marionette automation API into its own package and not have a monolithic package	David Burns :automatedtester
906712	P1	implement tab modal dialog handling for desktop with Marionette (alert, confirm, prompt)	Chris Manchester (limited bugmail, email directly)
940954	P1	Let Marionette support shutdown/restart tests	David Burns :automatedtester
941749	P1	getWindows should return tabs	Chris Manchester (limited bugmail, email directly)
1169180	P1	Intermittent failure for update tests and unit tests in test_about_window.py due to Marionette exception: "win.document.documentElement is null"	Chris Manchester (limited bugmail, email directly)
1129702	P1	Add support for doubleclicking to marionette's actions	Chris Manchester (limited bugmail, email directly)
1106913	P1	Detect cyclic object references when marshaling objects in evaluate.toJSON	Andreas Tolfsen ❲:ato❳
1103196	P1	Support for untrusted/self-signed TLS certificates	Andreas Tolfsen ❲:ato❳
1097705	P1	Add ability to right or middle click an element	Chris Manchester (limited bugmail, email directly)
1046512	P3	Marionette breaks in e10s enabled Nightly	Chris Manchester (limited bugmail, email directly)
1088116	--	Marionette support for testing l10n builds at parity with mozmill	Chris Manchester (limited bugmail, email directly)
1091662	--	Ensure shortcuts work in Marionette tests	Chris Manchester (limited bugmail, email directly)
1092223	--	Marionette tests fail with e10s enabled
1093707	--	Using navigate() in chrome scope locks up the browser	Chris Manchester (limited bugmail, email directly)
1094441	--	Enable 'css selector' strategy in chrome scope	David Burns :automatedtester
1096488	--	Marionette support for navigating between remote and non-remote pages with e10s enabled	Chris Manchester (limited bugmail, email directly)
1096571	--	Don't call into Webdriver's isShown() method when attempting to manipulate content xul elements	Andrew Halberstadt [:ahal]
1084412	--	searchTimeout doesn't work when using findElements	Andrew Halberstadt [:ahal]
1099331	--	Marionette's navigate errors on loading an about:blocked or about:error page	Chris Manchester (limited bugmail, email directly)
1083131	--	Marionette does not always remove temporary profiles	Chris Manchester (limited bugmail, email directly)
1104742	--	Stopping Marionette with Ctrl-C does not quit the application	Andrew Halberstadt [:ahal]
1080772	--	Convert a few mozmill tests to Marionette as POC	Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)
755036	--	Save globals between execute_script calls in 'chrome' contexts	Chris Manchester (limited bugmail, email directly)
1114623	--	Implement closeChromeWindow() to close the currently active chrome window	Chris Manchester (limited bugmail, email directly)
1120630	--	Turn on Mn-e10s	Chris Manchester (limited bugmail, email directly)
1120809	--	Marionette sometimes fails to register a content script with e10s enabled	Chris Manchester (limited bugmail, email directly)
1122609	--	execute_script() called with new_sandbox=False in chrome context does not use current script_args but those from a former call	Chris Manchester (limited bugmail, email directly)
1123683	--	Set default testing preferences (e.g. disabling FHR and other metric pings)	Barbara Miller (:galgeek)
1124604	--	Add `focus` parameter to switch_to_window() to stay in compat with switch_to_frame()	Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)
1124667	--	Release marionette-driver 0.2 and marionette-client 0.9	Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)
1125209	--	Sending a string with capital letters should not trigger a shift key
1125377	--	Marionette fails during getWindowHandles with e10s turned on when accessing a CPOW	Chris Manchester (limited bugmail, email directly)
1080764	--	Add support for anonymous nodes in chrome	Andrew Halberstadt [:ahal]
1135846	--	Re-organize marionette actions code to expose actions to chrome scope	Chris Manchester (limited bugmail, email directly)
1137388	--	Marionette hard-kills the browser for restarts which causes lots of trouble	Chris Manchester (limited bugmail, email directly)
1141483	--	After a restart Marionette doesn't restore the previous context	Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)
1141519	--	Marionette hangs if exceptions are happening in content scope	David Burns :automatedtester
1141679	--	Marionette doesn't clean up its socket following a restart initiated inside the application	Chris Manchester (limited bugmail, email directly)
1144907	--	Marionette's gather_debug shouldn't leave calling code in content scope if it didn't start there	Chris Manchester (limited bugmail, email directly)
1028407	--	Try to make marionette connect to browser sessions quicker	Chris Manchester (limited bugmail, email directly)
1169600	--	"MarionetteException: this.browserForTab is null" when closing the first tab of a chrome window	Chris Manchester (limited bugmail, email directly)
1169798	--	MarionetteException in switch_to_window() with chrome scope enabled: "this.browser is undefined"	Chris Manchester (limited bugmail, email directly)
1176663	--	[e10s] Marionette.navigate() hangs when first tab gets closed with: NS_ERROR_ILLEGAL_VALUE in nsIMessageSender.sendAsyncMessage
1176758	--	Allow mozprocess to detect and kill detached child processes	Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)
1202392	--	Improve exception handling and message details in Marionette	Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)

45 Total; 45 Open (100%); 0 Resolved (0%); 0 Verified (0%);