Confirmed users
3,376
edits
Auto-tools/Projects/Stockwell/backfill-retrigger: Difference between revisions
Jump to navigation
Jump to search
m
Auto-tools/Projects/Stockwell/backfill-retrigger (view source)
Revision as of 17:11, 14 March 2018
, 14 March 2018→how many retriggers: initial data
m (→how to find which job to retrigger: initial data) |
m (→how many retriggers: initial data) |
||
| Line 31: | Line 31: | ||
= how many retriggers = | = how many retriggers = | ||
There is not an exact science here, but I typically choose 40 as my target. This allows us to easily see a pattern and find a percentage of pass/fail jobs per revision. If a test is failing 10% of the time, we would need 10 data points to see 1 failure. Imagine if we had 18 green tests and 2 failed tests, it is often wise to get at least 20 data points. I choose 40 because then in the case of 10% failure rate we have more evidence to show 10% failure rate. | |||
If we look at the data from orangefactor it cannot tell us a real failure rate. The data we have will say X failures in Y pushes (30 failures in 150 pushes). While that looks like a 20% failure rate, it can be misleading for a few reasons: | |||
* we do not run every job/chunk on every push, so it could be 30 failures in 75 data points | |||
* there could be retriggers on the existing data and we could have 3 or 4 failures on a few pushes making it failing less than 20% | |||
= what to do with the data = | = what to do with the data = | ||
= exceptions = | = exceptions = | ||