canmove, Confirmed users
1,126
edits
Sheriffing/Job Visibility Policy: Difference between revisions
Add mention of test timebombs & link to the MDN "Avoiding intermittent oranges" guide
(→3) Runs on all trees that merge into mozilla-central: Clarify "all trees that merge into mozilla-central") |
(Add mention of test timebombs & link to the MDN "Avoiding intermittent oranges" guide) |
||
| Line 45: | Line 45: | ||
* A mozilla-central push results in ~400 jobs. A 5% failure rate would mean 20 failures on that push - ie: an OrangeFactor of 20. The typical OrangeFactor across all trunk trees is normally 3-5, so a 5% failure rate is extremely generous. | * A mozilla-central push results in ~400 jobs. A 5% failure rate would mean 20 failures on that push - ie: an OrangeFactor of 20. The typical OrangeFactor across all trunk trees is normally 3-5, so a 5% failure rate is extremely generous. | ||
=== 8) Supports the disabling of individual tests === | === 8) Must avoid patterns known to cause non deterministic failures === | ||
* Must not rely on resources outside of the build network: | |||
** Since these will cause failures when the external site is unavailable, as well as impacting end to end times & adding noise to performance tests. | |||
** eg: Emulator/driver binaries direct from a vendor's site, package downloads from PyPi or page assets for unit/performance tests. | |||
* Must not contain time bombs, e.g. tests that will fail after a | |||
certain date or when run at certain times. | |||
* See the [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding_intermittent_oranges guide on avoiding intermittent failures]. | |||
=== 9) Supports the disabling of individual tests === | |||
* It must be possible for sheriffs to disable an individual test per platform or entirely, by either annotating the test or editing a manifest/moz.build/Makefile. (See also requirement #10). | * It must be possible for sheriffs to disable an individual test per platform or entirely, by either annotating the test or editing a manifest/moz.build/Makefile. (See also requirement #10). | ||
=== 10) Has sufficient documentation === | === 10) Has sufficient documentation === | ||
| Line 62: | Line 66: | ||
=== 11) Easy for a dev to run locally === | === 11) Easy for a dev to run locally === | ||
* Is supported by mach. | * Is supported by mach. | ||
* Ideally part of mozilla-central (legacy exception being Talos). | * Ideally part of mozilla-central (legacy exception being Talos). | ||
== Requesting changes in visibility == | == Requesting changes in visibility == | ||