Partnering:Projects:BYOB:BuildSystemDesign

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

Build system design

This document contains design specifications for the BYOB build system.

We're not doing anything horribly complex here, so rather than reinvent the wheel, I recommend we go with a simple buildbot master/slave setup with custom hg pollers to trigger

Why use buildbot?

• open source with an active development community
• leverage existing institutional knowledge in the RelEng team

Master setup

The buildbot master can run either Mac or Linux.

Based on RelEng experience with buildbot masters, a physical machine is recommended *if* we project an order-of-magnitude increase in BYOB submissions. At present, we're unlikely to see too much load (comparable with our core build systems), so a VM would likely suffice.

Slave setup

Due to Mac packaging requirements, Mac slaves are required. These can be physical machines (recycled minis perhaps?) or VMs if the OS X VMs prove performant enough.

If we are going to due our own signing for BYOB builds, we will also need a Windows slave. Again, this slave can be a physical machine or a VM.

If we plan to offer BYOB for mobile, we'll need to install the appropriate Debian packaging tools for Mac.

Depending on projected demand for the BYOB service, 3-5 slaves would be a good initial investment.

Repack hosting and storage

We'll need a file server where we can stage the in-progress (unsigned) and completed repacks. I defer any decisions here to IT.

Required customizations

Assumptions:

• configurations will be stored in a version control repository (Mercurial)
• Windows repacks need to be signed, so the repack process for Windows builds will differ slightly than for other platforms. Windows repacks will initially end up in a temporary dir pending signing.

We can setup a number of different custom triggers/hg pollers depending on how we want to generate builds. Here's a list of the triggers I think we need:

• build per check-in: triggers all builds changed by the check-in. Builds are broken up by repack so that they can be parallelized across the build slaves. We'll have a suitable idle timer in place to catch "oops, i missed one thing"-type errors.
• one-off/single builds: can be triggered from the buildbot master directly, but we could also hook up a sendchange request to the webtool to accomplish the same thing.
• re-generate all active BYOB repacks: this will likely be a custom sendchange that will generate a job per repack. This will be useful for security releases.
• signing poller: looks for builds in an incoming directory, signs them, puts them into the final location.