View source for ReleaseEngineering/BuildFarmRefresh/2012Q2

__TOC__
= Platform priority =
The following platform priority list is from a recent [[Firefox/Channels/Meetings/2012-05-08#RelEng|Channel meeting]], but has been extended to include existing platforms and new platforms that are already in-flight: 
* OS X 10.8 signing of 10.7 builds
* unified Windows builders (single 64bit image that can create Win8, win64, win32-on-win64 builds to prevent linker issues in the future)
* Windows 8 build hardware
* Windows 8 test hardware
* b2g
* unified Linux builders (single 64bit image+mock)
* Rapid Betas
* mobile test hardware (pandaboards)
* builds for armv6
* builds for android-on-intel 
* Mac 10.8 test hardware
* Mac 10.8 build hardware?
* new Windows test hardware
* new Linux test hardware

= Major goals =
* tackle many targets in parallel rather than serially as in the past
** requires investing more than one human resource
* virtualize as much hardware as is possible
** caveats:
*** only works for building and unittests(correctness), not talos (performance)
*** build times must be competitive with hardware: 
**** perf: 25% slower may be acceptable, order-of-magnitude is not
**** cost: costs of bulk AWS VMs? costs of VMs (incl ESX lic.), netapps, hosts?
*** should VMs be able to run on both AWS and Mozilla hardware?
* Queue collapsing
** currently using queue collapsing to mitigate lack of capacity.
** should have enough capacity that queue collapsing is again considered an error condition; setup nagios alerts when queue collapsing triggered.

= Plan of attack =
# acquire signing key from Apple
#* releng: joduinn, catlee
# setup 10.8 signing machine
#* https://bugzilla.mozilla.org/show_bug.cgi?id=744789
#* releng: bhearsum
# segregate 5 Win64 build machines to be exclusively used as Win8 builders until a unified image is available
#* unblocks developers immediately
#* tools can be installed by hand until GPO is ready
#* releng: armenzg; IT: MaRu
#* https://bugzilla.mozilla.org/show_bug.cgi?id=754008
#* https://bugzilla.mozilla.org/show_bug.cgi?id=737994
# create single linux64 VM that can build all current linux-based builds: linux32, linux64, android*, b2g
#* mock envs for each build type
#** catlee already has an image on AWS that does almost all of this AIUI. Still many services issues to be worked out there though.
#* based on something more recent than CentOS 5 (CentOS 6?) 
#* virtualizing b2g builders frees up DL120 hardware for use as Win8 testers
#* virtualizing all linux builders frees up ix machines for use as Windows builders
#* releng: catlee, bear; IT: arr
# create single Win64 hardware image that can build all current Windows-based builds: Win32, Win64, Win8
#* image all ix machines with new Win64 image
#* releng: catlee, armenzg; IT: MaRu
#* https://bugzilla.mozilla.org/show_bug.cgi?id=753132
# setup DL120s as Win8 testers
#* buy graphics cards for existing DL120s (have 3, need 50)
#* determine how many more systems we need to purchase
# rackmount enclosure for pandaboards
#* still blocked on initial automation work by A-Team
# create mock env for armv6
# create mock env for android-on-intel

= Open questions =
* switching (back!) to VMs for unittests
** switching unittests from VMs to hardware reduce intermittent oranges.
** is this practical?
* where should VM capacity come from (AWS vs Mozilla IT vs ???)
* capacity for existing platforms. Do we need more:
** tegras?
** rev5 builders?
** rev5 testers?
* new hardware for general Linux and Windows testing?
** same hardware between platforms? (no requirement that it be the same from the dev side)
** DL120s?
** how many, and for which OSes?