WebExtensions/Try Server: Difference between revisions

← Older edit

WebExtensions/Try Server (view source)

Revision as of 09:30, 3 October 2023

1,969 bytes added , 3 October 2023

Add mach try fuzzy

Robwu

44

edits

@@ Line 1: / Line 1: @@
 = Try Server for WebExtensions =
 WebExtension code is covered by unit tests, which are described at [[WebExtensions/Hacking#Unit_tests]] and [[WebExtensions/Contribution_Onramp#Testing]]. These tests can be run on the [[ReleaseEngineering/TryServer#Try_Server|Try Server]]. Contributors [[ReleaseEngineering/TryServer#Getting_access_to_the_Try_Server|with access to the try server]] can [[ReleaseEngineering/TryServer#How_to_push_to_try|push try jobs]] to run tests.
+For up-to-date documentation about the try server, see [https://firefox-source-docs.mozilla.org/tools/try/index.html Pushing to Try in the Firefox Source Docs].
 Try jobs can be scheduled via the web interface or via the "mach try" command.
@@ Line 8: / Line 11: @@
 * linux,linux64
 * macosx64
-* android-api-16,android-x86 (android-x86 is faster)
+* android-x86
-Symbols for the following [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing#In_production test suites] are shown in the test results on Treeherder. Test suite identifiers for use with "-t" in "mach try" are listed below the main bullet points:
+Symbols for the following [https://web.archive.org/web/20191005073152/https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing#In_production test suites] are shown in the test results on Treeherder. Test suite identifiers for use with "-t" in "mach try" are listed below the main bullet points:
-* M (mochitests) and M-e10s (mochitests in multi-process mode)
+* M (mochitests) and M-1proc (mochitests in single-process mode)
-** mochitests (all mochitests, including many non-extension test suites). Not to be confused with "mochitest" (which only runs plain non-e10s mochitests).
+** mochitests (all mochitests, including many non-extension test suites). Not to be confused with "mochitest" (which only runs plain e10s mochitests).
-** mochitest-bc ([https://developer.mozilla.org/en-US/docs/Mozilla/Browser_chrome_tests browser chrome tests]; desktop only - Windows, Linux and macOS)
+** mochitest-bc (browser chrome mochitests, i.e. test/browser/*.js; desktop only - Windows, Linux and macOS)
-** mochitest-o (Android only)
+** mochitest ([https://firefox-source-docs.mozilla.org/testing/mochitest-plain/index.html plain mochitests], i.e. test/mochitest/*.html)
-* X (xpcshell tests)
+* X (xpcshell tests) and X-1proc (xpcshell tests in single-process mode)
 ** xpcshell
-* TV ([https://developer.mozilla.org/en-US/docs/Mozilla/QA/Test_Verification test verification mode]; same effect as "mach test path/to/your/test --verify" locally) (desktop platforms only; not available on Android)
+* TV ([https://firefox-source-docs.mozilla.org/testing/test-verification/index.html test verification mode]; same effect as "mach test path/to/your/test --verify" locally) (desktop platforms only; not available on Android)
-** test-verify-e10s
+** test-verify
-* ES ([https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/ESLint ESlint])
+* ES ([https://firefox-source-docs.mozilla.org/code-quality/lint/linters/eslint.html ESlint])
 Pass --artifact to speed up builds (only if your changes do not require recompilation of C++/Rust code, such as Java (Android) and JavaScript-only changes).
@@ Line 32: / Line 35: @@
 == Examples ==
-Examples of "mach try" commands (add "--no-push" to see the generated try syntax without actually triggering the try job):
+By default, "mach try" is an alias for [https://firefox-source-docs.mozilla.org/tools/try/selectors/auto.html "mach try auto"], which attempts to select some relevant tests. If unsure about what to run, "mach try" is recommended.
+To have more control over the tests to run, the newer [https://firefox-source-docs.mozilla.org/tools/try/selectors/fuzzy.html "mach try fuzzy"] or the classic [https://firefox-source-docs.mozilla.org/tools/try/selectors/syntax.html "mach try syntax"] can be used instead.
+=== mach try fuzzy ===
+Examples of "mach try fuzzy" commands (add "--no-push" to see the selected jobs without submitting to try):
+* Run most relevant WebExtensions tests. This is recommended and was added in https://bugzilla.mozilla.org/show_bug.cgi?id=1856473
+  <nowiki>./mach try --preset webextensions</nowiki>
+* Run relevant extension API tests on Android:
+  <nowiki>./mach try fuzzy -q "'geckoview '64-qr !gpu mobile/android/components/extensions toolkit/components/extensions</nowiki>
+* (Note: if you get <nowiki>bash: !gpu: event not found</nowiki>, replace all exclamation marks (<nowiki>!</nowiki>) with <nowiki>"'!'"</nowiki>)
+* Run relevant extension API and AOM tests on Windows:
+  <nowiki>./mach try fuzzy -q "'windows11-64-2009-qr !wpt !gpu !msix browser/components/extensions toolkit/components/extensions toolkit/mozapps/extensions</nowiki>
+If in doubt about what to use in the query (<nowiki>-q</nowiki>), run <nowiki>./mach try fuzzy --no-push</nowiki> to launch an interactive program that displays the matched jobs.
+=== mach try syntax ===
+Examples of "mach try syntax" commands (add "--no-push" to see the generated try syntax without actually triggering the try job):
 * Run all relevant tests. This is expensive and should only be used when you don't know which tests should be run, because it also includes many non-extension tests.
-   <nowiki>mach try -b do -p win64,linux64,macosx64,android-api-16 -u mochitests,xpcshell,test-verify-e10s --artifact</nowiki>
+   <nowiki>mach try syntax -b do -p win64,linux64,macosx64,android-x86 -u mochitests,xpcshell,test-verify --artifact</nowiki>
 * Run all relevant tests, without non-extension tests. The command with the `--tag=webextensions` flag uses your local checkout and .mozconfig configuration to identify the tests. Some Android-only tests may not run if your current configuration targets desktop.
-   <nowiki>mach try -b do -p win64,linux64,macosx64,android-api-16 -u test-verify-e10s --artifact --tag=webextensions</nowiki>
+   <nowiki>mach try syntax -b do -p win64,linux64,macosx64,android-x86 -u test-verify --artifact --tag=webextensions</nowiki>
-* Run browser-chrome extension tests (and many other browser-chrome tests, but at least less expensive than just "mochitests"). This test type is only used on desktop, so Android was omitted from the platform flag ("-p").
+* Run most extension tests (xpcshell, plain mochitests and browser-chrome extension tests (and many other mochitests/browser-chrome tests, but at least less expensive than just "mochitests"). The mochitest-bc test type is only used on desktop, so Android was omitted from the platform flag ("-p").
-   <nowiki>mach try -b do -p win64,linux64,macosx64 -u mochitest-bc,test-verify-e10s --artifact</nowiki>
+   <nowiki>mach try syntax -b do -p win64,linux64,macosx64 -u xpcshell,mochitest,mochitest-bc,test-verify --artifact</nowiki>
 * Run extension tests on Android (requires a [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_for_Android_build#Prepare_a_Firefox_for_Android_mozconfig_file mozconfig file for Android]; otherwise tests in mobile/android/components/extensions/test/ will not be run. The MOZCONFIG environment variable is used to select a different mozconfig file, so that you can develop for multiple platforms with a single checkout):
-   <nowiki>MOZCONFIG=/path/to/mozconfig-for-android mach try -b do -p android-api-16 --artifact --tag=webextensions</nowiki>
+   <nowiki>MOZCONFIG=/path/to/mozconfig-for-android mach try syntax -b do -p android-x86 --artifact --tag=webextensions</nowiki>
-It may take several seconds before a "mach try" command with the "--tag=webextensions" flag completes. If you already know the directory that contains the test, it is recommended to use the directory instead of "--tag=webextensions", e.g.:
+It may take several seconds before a "mach try syntax" command with the "--tag=webextensions" flag completes. If you already know the directory that contains the test, it is recommended to use the directory instead of "--tag=webextensions", e.g.:
 * Run tests in specific directories for desktop:
-   <nowiki>mach try -b do -p win64,linux64,macosx64 -u mochitests,mochitest-bc,xpcshell --artifact --and toolkit/components/extensions/test/ browser/components/extensions/test/</nowiki>
+   <nowiki>mach try syntax -b do -p win64,linux64,macosx64 -u mochitests,mochitest-bc,xpcshell --artifact --and toolkit/components/extensions/test/ browser/components/extensions/test/</nowiki>
 * Run tests in specific directories for Android:
-   <nowiki>MOZCONFIG=/path/to/mozconfig-for-android mach try -b do -p android-api-16 -u mochitests,mochitest-o,xpcshell --artifact --and toolkit/components/extensions/test/ mobile/android/components/extensions/test/</nowiki>
+   <nowiki>MOZCONFIG=/path/to/mozconfig-for-android mach try syntax -b do -p android-x86 -u mochitests,xpcshell --artifact --and toolkit/components/extensions/test/ mobile/android/components/extensions/test/</nowiki>
 By default, a test that match any specified test tags ("--tag") or suite ("-u") will be scheduled. To only run tests that match all specified rules, include "--and" in the try syntax.