User:Mwobensmith/Security Bug Regression
Jump to navigation
Jump to search
Problem: We can't verify all fixed security bugs.
Reasons:
- We don't have time.
- We don't have builds that we need.
- ASan builds more than one month old
- ASan builds for things we currently only build as release builds (release, beta, ESR)
Current scenario:
- We verify some bugs - the most critical and/or testable - and not the rest.
Best case scenario:
- Verify all fixed security bugs, both on broken and fixed builds, and ensure test case lands in a test suite.
Worst case scenario:
- No verification of anything.
Some possible solutions:
- Get more ASan builds for releases where we previously don't have them.
- Get ASan builds on a more frequent basis to determine regression range. (?)
- Get a bigger repository that can hold more than one month of ASan builds (even for just the ASan builds we currently support).
- Create an automated or streamlined way to land bug files into test suites, thus reducing the need for manual regression.
- As above, create a tool like JSBugMon for ASan bugs.
- Strict enforcement of test cases for security bugs.
- Do what Google does. (?)
Other issues:
- QA insists on verifying broken build for a given bug, yet JS bugs are never treated this way by the current automated process. Is the risk of skipping this step worth the gains wed make in bug regression? How would this work in an automated system?
- Related issue of old JS bugs in Bugzilla whose tests have never landed in test suites - up to 2500 of them.
- Any automated solutions need to be runnable by others, can integrate/scale as needed. Avoid specialized solutions.
- We are trying to solve issues with bug regression, but there may be related problems of tracking and finding regression range.