Build:ClobberingATinderbox

In order to reduce interruptions and allow developers to clobber Tinderbox builds, we've deployed what we're calling "CLOBBER support" to the Mozilla Build Farm.

This allows developers to request clobber builds on various Tinderboxen.

Warning: This functionality is not yet available for Mercurial-based Tinderboxes, though it still works with CVS-based ones such as for the 1.9.0 (Firefox 3) and 1.8.1 (Firefox 2) branches. See "Caveats" below for details about working with Mercurial-based Tinderboxes.

Common reasons for clobbers include:

• Moving files/directories in ways that cause the dependency system to break
• Needing to respin a nightly build that has an obvious defect that prevents the nightly from being used

NOTE: clobbering a tinderbox will also clobber the corresponding unit test machine, i.e. you do *not* also have to clobber the buildbot machine. This is useful when code changes (e.g. dependencies) require all similar tinderboxen to be restarted. It is less useful when a single machine is simply hung.

To clobber a Tinderbox

1. After your code goes in...

2. Checkout the relevant branch/directory of mozilla/tools/tinderbox-configs.

To find the relevant branch, go to the last Tinderbox log of the build you want to clobber; near where the mozconfig is printed out, the versions and branches of the Tinderbox configs are also printed out. An example from a 1.8 build tinderbox:

USERNAME=cltbld
USERPROFILE=C:\Documents and Settings\cltbld
WINDIR=C:\WINDOWS
XPCOM_DEBUG_BREAK=warn
_=/bin/sh
__COMPAT_LAYER=EnableNXShowUI 
-->Tinderbox Config Info<--------------------------
Begin: Mon Feb 12 17:41:39 2007
cvs stat
cvs status: Examining .
===================================================================
File: mozconfig        	Status: Up-to-date

  Working revision:	1.2.6.1
  Repository revision:	1.2.6.1	/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/mozconfig,v
  Sticky Tag:		MOZILLA_1_8_BRANCH (branch: 1.2.6)
  Sticky Date:		(none)
  Sticky Options:	(none)

===================================================================
File: tinder-config.pl 	Status: Up-to-date

  Working revision:	1.2.10.1
  Repository revision:	1.2.10.1 /cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v
  Sticky Tag:		MOZILLA_1_8_BRANCH (branch: 1.2.10)
  Sticky Date:		(none)
  Sticky Options:	(none)

End:   Mon Feb 12 17:41:40 2007
-->END Tinderbox Configuration Information<--------------
-->mozconfig<----------------------------------------
#
## hostname: pacifica-vm
## uname: WINNT PACIFICA-VM 5.2 3790 xx I386
etc.

So, in the above case, you'd want to check out mozilla/tools/tinderbox-configs/firefox/win32 on the MOZILLA_1_8_BRANCH.

3. If there's not a CLOBBER file in that directory (case does matter), cvs add one. Edit the file with the reason (and hopefully a bug number?) for the clobber request.

4. Check it back in; include the reason in the commit message (yes, this is duplicated from the step above)

5. Annotate the build on Tinderbox to let others know it will (should be ;-) a clobber build, and thus will take longer to complete.

6. If you're feeling benevolent, please stop by #build to let us know you requested a clobber, so we know to expect it

7. Next build should be a clobber build; if it's not, please stop by #build to let us know, so we can track down why it didn't work

To clobber a unit test machine

Same as the above for tinderbox, but the CLOBBER file you need to create/update lives in a different directory in CVS. Check out the following module:

mozilla/tools/buildbot-configs/testing/unittest/CLOBBER

You'll find the following directory structure in the CLOBBER module:

<product>/<branch>/<platform>/

To clobber the , e.g., Win2k3 unit test box, you would update the following CLOBBER file:

CLOBBER/firefox/TRUNK/win2k3/CLOBBER

Caveats

Obviously, clobber builds take longer than depend builds, so please only use this functionality when necessary; also, please coordinate clobbering on IRC, so builds aren't clobbered multiple times for the same reason. To help mitigate, it might be useful to have a sheriff check in a clobber request or in the case of a CVS checkin that you know will break Tinderbox, the committer.

Due to deficiencies in Tinderbox, some Tinderboxen will not comply with clobber requests. Tinderboxen that are likely to not obey are those which do not produce nightly builds of any kind (fxdbug-win32-tbox, for example).

We are currently working through making it so all Tinderboxen pay attention to clobber requests. If you find one that does not, please let us know, so we can add it to the migration list.

If you need to clobber a Mercurial-based tinderbox, you should file a ticket. bug 432236 tracks adding the self-service CLOBBER capability to the mozilla-central builds.