ReleaseEngineering/Preproduction/Intro
The preproduction system is a copy of the production system, has its own pool of slaves, stage, aus2 and tinderbox trees. There is a "manager master" which monitors releng repositories, runs code quality checks, and if all of the needed checks pass, reconfigures the "slave masters" - scheduler and builder masters - our usual master types.
Goals
There are 2 main goals of preproduction system:
Testing the code before landing it to production
- One of the problems of the current downtime process is that the build duty person needs to check a bunch of patches and they may be conflicting (not apply properly in other words) or even worse, one patch may change the behavior of another one.
- The patches sometimes break master reconfig process even though we have test-masters.sh script but forget to run it (we aren't robots, are we?!)
- Even if the test-masters.sh script has been passed there is a possibility to get run time exceptions
To eliminate the problems of code quality we run some tests for every check in (like test-masters.sh, pylint, coverage). Additionally run usual builds and gather runtime exceptions as we do for production.
Simplifying the downtime process
There is a possibility of getting bitrotten patches during checking them in during the downtime.
To eliminate this problem should change the downtime process.
- Every check in will go to the default branch.
- Instead of check in patches during the downtime, build duty person will merge already checked in patches to the "production" branch. As a result we won't have problems with applying patches. Well almost won't have.
- We also may want to have some kind of quarantine for the check ins to be merged to the production branch. ("Don't merge check ins with age less than 24 hours").
- So, as a result, we'll have code tested not only during the downtime and the downtime itself will become a little bit less painful.
Current status
The initial preproduction system is up and running here: http://preproduction-master.build.mozilla.org:8710/
And the corresponding repo is here: http://hg.mozilla.org/build/preproduction/