User:Mwobensmith/Security Bug Regression: Difference between revisions
Jump to navigation
Jump to search
Mwobensmith (talk | contribs) No edit summary |
Mwobensmith (talk | contribs) No edit summary |
||
| Line 7: | Line 7: | ||
** ASan builds more than one month old | ** ASan builds more than one month old | ||
** ASan builds for things we currently only build as release builds (release, beta, ESR) | ** ASan builds for things we currently only build as release builds (release, beta, ESR) | ||
Current scenario: | Current scenario: | ||
| Line 17: | Line 16: | ||
Worst case scenario: | Worst case scenario: | ||
* No verification of anything. | * No verification of anything. | ||
Some possible solutions: | Some possible solutions: | ||
| Line 28: | Line 25: | ||
* Strict enforcement of test cases for security bugs. | * Strict enforcement of test cases for security bugs. | ||
* Do what Google does. (?) | * Do what Google does. (?) | ||
Other issues: | Other issues: | ||
| Line 34: | Line 30: | ||
* Related issue of old JS bugs in Bugzilla whose tests have never landed in test suites * up to 2500 of them. | * Related issue of old JS bugs in Bugzilla whose tests have never landed in test suites * up to 2500 of them. | ||
* Any automated solutions need to be runnable by others, can integrate/scale as needed. Avoid specialized solutions. | * Any automated solutions need to be runnable by others, can integrate/scale as needed. Avoid specialized solutions. | ||
* We are trying to solve issues with bug regression, but there may be related problems of tracking and finding regression range. | |||
Revision as of 00:31, 22 December 2012
Problem: We can't verify all fixed security bugs.
Reasons:
- We don't have time.
- We don't have builds that we need.
- ASan builds more than one month old
- ASan builds for things we currently only build as release builds (release, beta, ESR)
Current scenario:
- We verify some bugs * the most critical and/or testable * and not the rest.
Best case scenario:
- Verify all fixed security bugs, both on broken and fixed builds, and ensure test case lands in a test suite.
Worst case scenario:
- No verification of anything.
Some possible solutions:
- Get more ASan builds for releases where we previously don't have them.
- Get ASan builds on a more frequent basis to determine regression range. (?)
- Get a bigger repository that can hold more than one month of ASan builds (even for just the ASan builds we currently support).
- Create an automated or streamlined way to land bug files into test suites, thus reducing the need for manual regression.
- As above, create a tool like JSBugMon for ASan bugs.
- Strict enforcement of test cases for security bugs.
- Do what Google does. (?)
Other issues:
- QA insists on verifying broken build for a given bug, yet JS bugs are never treated this way by the current automated process. Is the risk of skipping this step worth the gains we'd make in bug regression?
- Related issue of old JS bugs in Bugzilla whose tests have never landed in test suites * up to 2500 of them.
- Any automated solutions need to be runnable by others, can integrate/scale as needed. Avoid specialized solutions.
- We are trying to solve issues with bug regression, but there may be related problems of tracking and finding regression range.