ReleaseEngineering/Maintenance

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Ready to rollout

Bug #	Description	machine name(s)	Urgency	Status	Need downtime?	Owner
bug 459269	setup places project branch	production and staging masters	medium	ready to land	no - but downtime would be better	lsblakk
bug 473821	add tp4 test to moz-central only (first step in full rollout of tp4)	qm-rhel02, talos-master	high	ready to land	YES	alice
bug 486614	Puppet on Linux (moz2-* machines)	moz2-linux-*	medium (q2 goal)	ready to go	no	bhearsum
bug 496712	rename try build dirs / Corrupt file system on WINCE & Windows hg builder? Requires some buildbot upgrade (per bhearsum in irc)	try master	low	ready for landing	yes	bhearsum

Upcoming changes

Bug #	Description	Master	Urgency	Status	Need downtime?
bug 480081	buildbotcustom changes for l10n nightly updates, toggled on for m-c and off for 1.9.1	production-master	low	backed out due to errors	no
bug 487329	graph server migration (switch from using graphs.m.o to graphs-new.m.o full time)	qm-rhel02, talos-master, production-master	medium	ready to land, waiting on blog post/IT assistance for fixing links broken with the migration	yes
bug 498715	Release builder prioritization	production-master	low	ready to land, tested in staging. requires buildbot master upgrade	yes
bug 499506 unpack.sh fails DMG unpacking when machine has slow I/O	use an unmount script based on catlee's work	anything running releases	low	ready for landing on tools	none
bug 493752	Reduce buildbot logging	production-master	low	ready to land, tested in staging	yes
bug 500246	reset TryServer repo	try-master	low		yes
unknown	netapp firmware update (joint goal w/IT)	unknown	unknown	unknown	no
bug 463262 automate nonsse unittest, being done manually for now	unknown	unknown	unknown	unknown	unknown
unknown	move to new faster keymaster (bug 496571 required first)	unknown	unknown	unknown	unknown
bug 383136 splitting out build-from-unittest	Requires bug 493755 (new installdmg.sh), and bug 448047	unknown	unknown	unknown	unknown
unknown	roll out ability to get crash drumps from frozen browser on winxp/vista	unknown	unknown	unknown	unknown
unknown	adding dirty profile test to talos	unknown	unknown	unknown	unknown
unknown	install sshd on all windows talos slaves	unknown	unknown	unknown	unknown

Previous changes

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Outcome	When	Bug #	Description	Master	Urgency	Status	reconfig/restart required
buildbotcustom portion backed out (because it was temporary)	20090629 0845 PDT	bug 500422	config bumps for 3.5 final tagging & source	production-master	high	landed	reconfig
In production	20090626	bug 483968 talos should be able to get a stack trace from browser hangs	fix for getting stack traces for frozen browsers on linux only	cvs talos check-in	low	landed	check-in
In production	20090626 0744 PDT	bug 488732	Do multiple unittest runs on a single build	production-master	low	landed	reconfig
In production	20090626 0744 PDT	bug 494622	Vista machines not rebooting reliably	talos-master	low	landed	reconfig
In production	20090626 0744 PDT	bug 488367	Running JSS in talos pool, migrating JSS slaves over.	talos master	low	landed	reconfig
In production	20090626 0744 PDT	bug 445611	Try server environment variables being set uniformly from env.py (2 patches, touches buildbotcustom so will affect production)	try master	low	landed	reconfig
In production	20090626 0744 PDT	bug 473259 /buildbotcustom/unittest/steps.py : update the rest of the code after bug 468023	re-re-land Serge's patch to tinderbox	production-master, etc	low-medium	landed	reconfig
In production	20090626 0744 PDT	bug 499398	Fix off-by-one error when grabbing changeset ID	try master	low	landed	reconfig
In production	20090624 0037 PDT	bug 499687	relauto config & l10n-changesets for Fx 3.5rc3 build2	production-master	high	landed	reconfig
In production	20090623 1608 PDT	bug 499687	relauto config & l10n-changesets for Fx 3.5rc3 build1	production-master	high	landed	reconfig
In production	20090619 1300 PDT	bug 498671	Fix start time reporting	production-master, talos-master	medium	ready to land	reconfig
In production, then backed out.	20090617 1000 PDT	bug 498701	comment out config file bumps for ReleaseUpdatesFactory so we can re-run	production-master	high	landed	reconfig
In production	20091616 2034 PDT	bug 498701	relauto config for Fx 3.5rc2 build2	production-master	high	landed	reconfig
In production	20091616 1809 PDT	bug 498701	relauto config for Fx 3.5rc2 build1	production-master	high	landed	reconfig
		bug 498757	update l10n-changesets for Fx 3.5rc2 build1	production-master	high	landed	reconfig
In production	20091616 0830 PDT	bug 495008	linux desktop fennec builds/l10n repacks	production-master	medium	landed	reconfig
In production	20090616 0830 PDT	bug 496509	Step name cleanup for build database	production-master	low	landed	reconfig
In production	20090616 0830 PDT	bug 495008	Desktop Maemo builds/repacks	production-master	low	landed	reconfig
In production	20090616 0830 PDT	bug 498188	mozconfig-wince should contain --enable-jemalloc	production-master	low	landed	reconfig
In production	20090616 0830 PDT	bug 471318	Remove index.cgi from tinderbox urls	production-master	low	landed	reconfig
In production	20090616 0830 PDT	bug 498468	Reduce l10n polling interval	production-master	low	landed	reconfig
In production	20090616 0830 PDT	bug 488368	Fetch symbols for nochrome talos tests on pool	talos-master	low	landed	reconfig
In production	20090615 1500 PDT	bug 497721	release automation config for fx3.5rc1 build2	production-master	high	landed	reconfig
In production	20090612 1100 PDT	bug 492808	Improve clobber messages	production-master	low	ready to land	reconfig
In production	20090612 1100 PDT	bug 497691	Fix quoting on Tinderboxprint statements	production-master	low	ready to land	reconfig
In production	20090612 1100 PDT	bug 448047	Fix symbol unpacking, and have talos print out which slave is running	talos-master	low	ready to land	reconfig
In production	20090612 1100 PDT	bug 496510	Sendchange to Talos should include revision	production-master	low	ready to land	reconfig
In production	20090612 0515 PDT	bug 497721	config bumps for 3.5rc1	production-master	high	landed	reconfig
In production	20090612 0515 PDT	bug 493637	updated l10n-changesets for 3.5rc1	production-master	high	landed	reconfig
In production	20090612 0515 PDT	bug 497778	use L10N_SLAVES for release builds	production-master	high	landed	reconfig
In production	20090610 1000 PDT	bug 495533	add leak threshold (for crashtest, this time) after java upgrade tripped small leak	production-master	high	landed	reconfig
In production	20090610 0700 PDT	bug 493755	installdmg fixes	production-master, talos-master	medium	ready to land	reconfig
In production	20090609 11:45pm PDT	bug 495533	add leak threshold after java upgrade tripped small leak	production-master	high	landed	reconfig
In production	20090609 9:40am PDT	bug 495610	Clean up whitespace in factory.py	production-master (already on staging)	low	landed	reconfig
In production	20090607 1955 PDT	bug 493957	fix references to 3.1b4 in automation config for 3.5b99	production-master	high	landed	reconfig
		bug 494325	setup bm-xserve11 for production	production-master	low	landed	reconfig
In production	20090605 13:30 PDT	bug 493957	Backout dummy tag patch and bump for 3.5b99build2	production-master	high	landed	reconfig
In production	20090605 0449 PDT	bug 493957	Dummy tag, source, en-US builds for 3.5 Preview (b99)	production-master	high	landed	reconfig
In production	20090604 2309 PDT	bug 493957	automation config for 3.5 Preview (b99)	production-master	high	landed	reconfig
		bug 493637	l10n-changesets for 3.5 Preview (b99)	production-master	high	landed	reconfig
In production	20090601 1051 PDT	bug 495778	all-locales logic change	l10n-server	low	landed	reconfig
In production	20090601 1051 PDT	bug 494376	set l10n_revision to actual revision	production	medium	landed	reconfig
In production	20090601 1045 PDT	bug 465039	Fix mobile try mail/upload dirs	try-master	low	landed	reconfig
In production	20090529 1420 PDT	bug 478436	Integrate compare-locales with l10n mercurial repack builds	production-master	medium	landed	reconfig
In production	20090529 0947 PDT	bug 465039	reenable tinderbox changeset links for mobile builds	production-master, try master	low	landed	reconfig
In production	20090528 1632 PDT	bug 465039	add wince, maemo try builds	production-master, try master	medium	landed	reconfig
Backed out	Backed out: 20090527 2057 PDT Landed: 20090527 1434 PDT	bug 473259 /buildbotcustom/unittest/steps.py : update the rest of the code after bug 468023	(Ev1a) Sync' Mochitest and Reftest: summarizeLog() and evaluateCommand()	production-master, etc	low-medium	re-landed	reconfig
In production	20090525 1400 PDT	bug 471318	removes unnecessary 'index.cgi' from tinderbox changeset urls	production-master	low	landed	reconfig
In production	20090525 1320 PDT	bug 488099	l10n/maemo repacks hang on 'echo None'	production-master	major	landed	reconfig
In production	20090524 1946 PDT	bug 494582	release repacks hang on 'echo None'	production-master	major	landed	reconfig
In production	20090522 1723 PDT	bug 494434	Maemo l10n repacks busted with linux-arm instead of linux-gnueabi-arm	production-master	major	landed	reconfig
In production	20090522 1010 PDT	bug 494367	Maemo l10n repacks busted with echo None	production-master	blocker	landed	reconfig
In production	20090522 0810 PDT	bug 494257	talos try broken for mac/win32 cvs builds	sm-try-master, qm-rhel02/tryperfmaster	medium	landed	reconfig
In production	20090522 0800 PDT	bug 494055	add xulrunner builders to post-stage dep scheduler	{staging,production}-1.9-master}	low	landed	reconfig
In production	20090522 0111 PDT	bug 494313	stop hiding Fx3.0 shark builds	production-1.9-master	low	landed	reconfig
In production	20090521 2051 PDT	bug 486454 bb-c:d7cd3e71df36 bbc:3901a3e9cd6e	Move update verify bumping to Updates step	production-master	medium	landed	reconfig
In production	20090521 0810 PDT	bug 488099	Expose build properties via TinderboxPrint	production-master	low	landed	reconfig
Backed out	20090521 0810 PDT	bug 473259 /buildbotcustom/unittest/steps.py : update the rest of the code after bug 468023	(Ev1a) Sync' Mochitest and Reftest: summarizeLog() and evaluateCommand()	production-master, etc	low-medium	backed out	reconfig
In production	20090521 0810 PDT	bug 491884	Remove create profile build step from unittest factory	production-master	low	landed	reconfig
In production	20090521 0810 PDT	bug 383136	Fix reboot step for unittest on packaged builds	production-master, staging-master	low (production machines don't reboot)	landed	reconfig
In production	20090521 0600 PDT	bug 435472	fix tinderbox link in try server e-mails	try-master	low	landed	reconfig
In production	20090520 1000 PDT	bug 481886	MozillaTryServerHgClone fixes for 0.7.10p1 + bustage fix in buildbot-configs (remove CreateProfile step from unittest runs)	try-master + slaves	medium	landed	restart
In production	20090519 0930 PDT	bug 435472	try server e-mail notification	try-master	medium	landed	restart
In production	20090519 0500 PDT	bug 490581	all builders should TinderboxPrint their slavenames	production-master	low	landed	reconfig
In production	20090515 0640 PDT	bug 383136	Unittests on packaged builds	production-master	medium	landed	reconfig
In production	20090515 0640 PDT	bug 492852	Allow branchName to be passed to factory	production-master	low	landed	reconfig
In production	20090515 0640 PDT	bug 492985	fix buildbotcustom.steps.l10n.setBuild usage of setProperty	no firefox ones	high	landed	n/a
In production	20090515 0640 PDT	bug 491884	remove CreateProfile step from unittest factory	production-master	low	landed	reconfig
In production	20090515 0640 PDT	bug 486567	unify try server code with other factories	try-master, production-master, try-talos	low	landed	reconfig
In production	20090514 1953 PDT	bug 489954 rev:3d4c6dde70a9	Missing complete mars for localised 3.5b4	production-master	low	landed	reconfig
Backed out after outage	20090512 1742 PDT / 20090513 0045 PDT	No bug	Set idle timers to 6 hours for EqualLogic outage	production-master	medium	landed	reconfig
In production	20090512 0915 PDT	bug 486540	more factory abstraction for comm-central	production-master	low	landed	reconfig
In production	20090512 0915 PDT	bug 492221	optional graph server posting	production-mastre	low	landed	reconfig
In production	20090511 12:40 PDT	bug 383136	Fix reftest parser	production-master, sm-try-master	high	landed	reconfig
In production	20090511 12:40 PDT	bug 383136	Minor fixes for unittests on packaged builds	production-master, staging-master	low	landed	reconfig
In production	20090511 1156 PDT	bug 462889	Enable maemo unittest packages	production-master	medium	landed	reconfig
In production	20090511 0645 PDT	bug 383136	Fix for new unittest steps that broke after buildbot upgrade. Listing here so that the next person to reconfig production-master doesn't freak out. This code isn't live on production-master, so should be safe.	staging-master	low	landed	reconfig
In production	20090511 0645 PDT	bug 482598	enable mac a11y tests on m-c and tracemonkey	production-master	medium	landed	reconfig
In production	20090506 1205 PDT	bug 489472	enable 1.9.1 mobile builds	production-master	medium	landed	reconfig
In production	20090504 0630 PDT	bug 487496	fix MozillaMochitest kwarg digging	production-master/try-master	low	landed	reconfig
In production	20090504 0630 PDT	bug 469518	enable reftest leak threshold	production-master/try-master	low	landed	reconfig
In production	20090504 0630 PDT	bug 480145	BuildStepStatus.setColor shouldn't be called from custom steps	production-master/try-master	low	landed	reconfig
In production	20090429 morning PDT	(no bug)	de7529bb1396 - try server env fix, 87500f280ecf - try server arg fix	try-master/staging-try-master	high	landed	reconfig
In production	20090429 0730 PDT	bug 487496	original buildbot-configs patch for talos + bustage fix	perfmaster/tryperfmaster/perfmaster-staging	high	landed	reconfig
In production	20090428 16:11 PDT	bug 490369	Add hg revisions to mobile build waterfall	production-master	Low	landed	reconfig
In production	20090425 19:30 PDT	bug 383136	unittests on packaged builds (enabled on staging only)	production-master, staging-master	medium	landed	reconfig
In production	20090424 9:30 PDT	bug 448047	Integrate perfrunner code into buildbotcustom	talos-master	low	landed	reconfig
In production	20090423 12:30 PDT	bug 488372	Notify new talos masters of builds	production-master	low	ready to land	reconfig
In production	20090422 13:30 PDT	bug 488411	Simplify slave listings	production-master	low	landed	reconfig
In production	20090422 13:30 PDT	bug 480524	Fennec l10n	production-master	medium	landed	reconfig
In production	20090422 13:30 PDT	bug 488349	Remove WinCE installer hacks	production-master	low	landed	reconfig
In production	20090421 10:45 PDT	bug 487496	buildstep parameter fixups	production-master	low	landed	reconfig
In production	20090420 17:05 PDT	bug 477792	hgpoller.py cleanup	production-master	low	landed	reconfig
In production	20090416 10:30 PDT	bug 488770	fix description in a couple of ShellCommands	production-master	low	landed	reconfig
In production	20090416 10:30 PDT	bug 487496	custom BuildStep's need to explicitly name arguments and call self.addFactoryArguments - buildbotcustom patch	production-master, try-master	low	landed	reconfig
In production	20090416 12:30 PDT	bug 488737	re-enable shark builds for 3.0.x	production-1.9-master	high	landed	reconfig
In production	20090416 12:30 pdt	no bug	bustage fix: use correct mac slave in production-1.9/master.cfg	production-1.9-master	high	landed	reconfig
In production	20090415 1648 PDT	bug 474544	Add bm-xserve12 to production roster	production-master	Low	landed	reconfig
In production	20090415 0745 PDT	bug 484203	Clobberer clobbered builds need to clearly identify themselves	production-master	Low	landed	reconfig
In production	20090414 0545 PDT	bug 487636	Create 6 new win32 buildbot slaves for Moz2/3.next pool - staging version	staging-master	Low	landed	reconfig
In production	20090414 01:55 PDT	bug 487636	Create 6 new win32 buildbot slaves for Moz2/3.next pool	production-master	Low	landed	reconfig
In production	20090413 15:05 PDT	bug 486513	move XULRunner builds to their own waterfall	production-master	Medium	landed	reconfig
		bug 476733 fix installer target	fix installer target	production-master	Medium	landed	reconfig
In production	20090413 13:45 PDT	bug 485736 Add (TUnit) 'xpcshell-tests' make target, using runxpcshelltests.py new '--manifest' option	(Ev1) Call the new target on tryserver	production-master	Medium (wanted on the road to stop double execution, which this creates :->)	landed	reconfig
In production	20090409 1130 PDT	bug 474666	Have code coverage builds report to tinderbox	production-master	low	landed	reconfig
In production	20090407 1315 PDT	n/a	Typo fix	production-master	low	landed (190b3af0a786)	reconfig
In production	20090407 1315 PDT	bug 485736 Add (TUnit) 'xpcshell-tests' make target, using runxpcshelltests.py new '--manifest' option	(Cv1a-FF) Call the new target	production-master	Medium (wanted on the road to run tests on packaged builds)	ready to land (on top of bug 479225 patches)	reconfig
In production	20090407 1315 PDT	bug 479225	(Ev1b) factory.py: Av4 leftovers after bug 445611 and bug 474666	production-master	Low	ready to land	reconfig
In production	20090407 1315 PDT	bug 479225	(Dv1a) steps.py: Av4 leftovers after bug 445611	production-master	Medium (wanted to remove workaround in SeaMonkey config)	ready to land	reconfig
In production	20090407 1315 PDT	bug 486111	use defaultdict for tree properties in l10n builds	production-master	Low (needed for l10n master v2, sync buildbotcustom)	ready to land, attachment 370873
In production	20090406 1908	bug 464014	Support rc's in hg relauto	production-master	Medium (ahead of 3.5b4)	landed in hg:buildbot-configs, hg:buildbotcustom	reconfig
		bug 486444	Broken win32_info.txt for releases	production-master	Medium (ahead of 3.5b4)	landed in hg:buildbotcustom	reconfig
In production	20090402 0943 PDT	bug 484542	prepare to merge buildrequests for l10n builds on 0.7.10+	production-master	Medium (wanted for l10n)	landed	reconfig
In production	20090401 1330 PDT	bug 486192	Don't try and match the end of line	production-master	Medium-High (fixes a regression)	hg:buildbotcustom	reconfig

Older Changes