Confirmed users
571
edits
Sheriffing/How To/Updating test expectations: Difference between revisions
Jump to navigation
Jump to search
Sheriffing/How To/Updating test expectations (view source)
Revision as of 15:02, 9 February 2022
, 9 February 2022add manual expectation updates for mochitests and web-platform-tests
(add reftests) |
(add manual expectation updates for mochitests and web-platform-tests) |
||
| Line 2: | Line 2: | ||
* web platform tests (wpt) are regularly imported from an external repository and the features tested might not have been implemented in Firefox yet. The import script sets the expectations based on Try pushes but the expected result can have changed because the feature got implemented between the Try push and the wpt test synchronization. | * web platform tests (wpt) are regularly imported from an external repository and the features tested might not have been implemented in Firefox yet. The import script sets the expectations based on Try pushes but the expected result can have changed because the feature got implemented between the Try push and the wpt test synchronization. | ||
* Tests developed by Mozilla are expected to pass but can be disabled for some platform configurations if they fail, e.g. new versions of operating systems often fail some tests and get added with tier 2 priority. The tests are initially skipped and bugs filed to fix the issue and reenable the test. | * Tests developed by Mozilla are expected to pass but can be disabled for some platform configurations if they fail, e.g. new versions of operating systems often fail some tests and get added with tier 2 priority. The tests are initially skipped and bugs filed to fix the issue and reenable the test. | ||
== mochitests == | |||
Mochitests test various parts of the browser, e.g. the UI functionality (browser-chrome, devtools), web browser interfaces (mochitest-plain). The most common annotation (below the test name in the <code>.ini</code> manifest file in the folder with the test file) is <code>skip-if =</code> followed by conditions under which the test shall not be executed, e.g. | |||
<code>skip-if = os == "win" && bits == 64</code> | |||
This skips the test for 64-bit builds on Windows. | |||
If the conditions are spread across multiple lines, fulfilling the conditions in one line causes the test to get skipped - the lines are connected by an "OR" condition. Example: | |||
<pre>skip-if = | |||
os == "win" && bits == 64 | |||
os == "linux"</pre> | |||
This causes the test to get skipped if the platform is either Windows with a 64-bit build or any Linux platform and Firefox build. | |||
* The variables available and their values are listed in each mochitest log near the end of the setup. Search for "These variables are available in the mozinfo environment and can be used to skip tests conditionally" in the log. | |||
== reftests == | == reftests == | ||
| Line 35: | Line 52: | ||
# Check <code>hg status</code> again. | # Check <code>hg status</code> again. | ||
# Save the changes with <code>hg commit -e</code> | # Save the changes with <code>hg commit -e</code> | ||
=== Manual update of expectations for web-platform-tests === | |||
See [https://firefox-source-docs.mozilla.org/web-platform/index.html?highlight=wpt#metadata the official documentation]. | |||