ReleaseEngineering/Maintenance

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

• buildbot
• buildbot-configs
• buildbotcustom
• tools
• talos

Ready to rollout

Are you changing the tool chain on a master? If so, let auto-tools know so they can update their masters

Bug #

Description

Master

Urgency

Status

Need downtime?

Owner

Previous changes

Did you change the tool chain on a master? If so, let auto-tools know so they can update their masters

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Outcome	When	Bug #	Description	Master	Urgency	Status	reconfig/restart required	Bug Owner
in production	20101118 1231 PST	bug 526495	DONTBUILD support	pm01 - scheduler master	low	landed	reconfig	catlee
in production	20101117 1855 PST	bug 574473	users/asasaki_mozilla.com/mozharness->build/mozharness	pm 8010/8011, pm02 8010, all builder_masters	medium	landed	restart	dustin
in production	20101117 1231 PST	bug 610045	Buildbot-0.8.2	all	low	landed	reconfig	catlee
in production	20101112 2020 PST	bug 589914, bug 608430	tag step a=release, android release fixes part 3	pm 8010/8011, pm02 8010	medium	landed	reconfig	aki
in production	20101112 1450 PST	bug 551756, bug 581106	fixes for shadow-central leak logs, turning on codesighs for s-c	builder_master*	low	landed	reconfig	lsblakk
in production	20101110 1558 PST	bug 609922	update to pageloader.xpi in b.m.o/talos/xpis	none	low	updated	no	lsblakk
in production	20101110 0833 PST	bug 571756	ignore cert for shadow-central download test/talos	tests_masters*	low	landed	reconfig	lsblakk
in production	20101110 0900 PST	bug 608269	dependencies for buildbotcustom	none	low	landed	no	dustin
in production	20101109 1020 PST	bug 551756	config changes for shadow-central upload urls	builder_master on pm01,pm03,sc01,sc02	medium	landed	reconfig	lsblakk
in production	20101108 1900 PST	bug 511282, bug 560236, bug 608747, bug 516984	fennec unit tests, slave name, more tegras, mozmill followup	all test-masters	medium	landed	reconfig	jhford,dustin,bear,jhford (jhford did reconfig)
in production	20101108 1600 PST	bug 607660, bug 610458, bug 610025	fix mobile buildsymbols, clean maemo objdir/dist/bin, c['changeHorizon'] = None	build scheduler, all builder masters, pm02	medium	landed	reconfig	aki, dustin
in production	20101105 1250 PDT	bug 570251	jetpack on win32	test-scheduler, all test-masters (including picking up mozmill change on tests_master5)	Low	landed	reconfig	lsblakk
in production	20101105 1135 PDT	bug 608163	fuzzing on win32	pm01, pm03, scheduler_master	Low	landed	reconfig	coop
in production	20101105 1015 PDT	bug 516984	mozmill on all platforms	all test masters (except tests_master5 which was missed) and test scheduler	medium	landed	reconfig	jhford
in production	20101104 0543 PDT	bug 608369	removed twinopen from Android Tegra test list	tm01-06, tests-scheduler	Low	landed	reconfig	bear
in production	20101103 1700 PDT	bug 511282	add linux mobile-desktop builders	all test masters	Low	landed	reconfig	jhford
in production	20101103 0600 PDT, 1253 PDT	bug 608369	add tpan, tsspider to Android Tegra test list	tm01, tests-scheduler at 12:53	Low	landed	reconfig	bear
in production	20101101 1130 PDT	bug 485519	sshd on windows build machines	n/a	low	landed	n/a	bhearsum/ssalbiz
in production	20101101 1130 PDT	bug 589885	hgtool support for shared source checkouts	n/a	low	landed		catlee/ssalbiz
in production	20101101 1130 PDT	bug 568035	Force /dev/ptmx permissions with puppet	puppet masters	low	landed		catlee
in production	20101101 1130 PDT	bug 596852	purge_builds should delete old directories	n/a	low	landed		catlee
in production	20101101 1130 PDT	bug 596744	Nightly jobs should happen only on fast machines	build masters	low	landed	reconfig	catlee
in production	20101101 1130 PDT	bug 606376	fuzzing output fix	build masters	low	landed	reconfig	catlee
in production	20101026 1930 PDT	bug 607240	firefox 3.5.15 automation config	pm02	high	landed	reconfig	bhearsum/nthomas
in production	20101026 1930 PDT	bug 607228	firefox 3.6.12 automation config	pm:8011	high	landed	reconfig	bhearsum/nthomas
in production	20101025 21:00 PDT	bug 600624, bug 603266, bug 601323	android multilocale/packaging/upload changes ported to releases; fix confvars.sh version bumping; fix desktop/android uploads	pm02(mobile)	high	landed	reconfig	aki
in production	20101023 12:10 PDT	bug 563382	android multilocale fixes	pm01,pm03,pm02(try),bm1,bm2	medium	landed	reconfig	aki
in production	20101022 15:00 PDT	bug 563382, bug 592704	android multilocale, sendchanges should check stdout	scheduler master, pm01,pm03,pm02(try and mobile),bm1,bm2	high	landed	reconfig	aki,gozer
in production	20101019 23:00 EDT, 2230 PDT	bug 605522	rename Tegra 250 to Android Tegra 250	test-master01, (tests-scheduler @ 2230)	high	landed	reconfig	bear
in production	20101019 23:00 EDT	bug 605541	add tinderboxprint of changeset details to output	test-master01	high	landed	restart	bear
in production	20101018 19:15 PDT	bug 605246	mochitest-a11y should not be run on Mac OS X 64 (debug and opt)	test-master02, talos-master02, tests_masters{3,4,5,6} and test-scheduler on pm02	bustage fix	landed	reconfig	lsblakk
in production	20101018 19:15 PDT	bug 605339	jetpack suite can't run without UnpackTests() to unzip the jetpack test dir	test-master02, talos-master02, tests_masters{3,4,5,6} and test-scheduler on pm02	low	landed	reconfig	lsblakk
in production	20101018 12:25 PDT	bug 603012	make env required arg in UnittestPackagedBuildFactory	all test/builder masters and schedulers	low	landed	reconfig	lsblakk
in production	20101018 12:25 PDT	bug 517475	stop running bloat steps on debug builds	all test/builder masters and schedulers	low	landed	reconfig	lsblakk
in production	20101018 12:25 PDT	bug 570251	Turning on jetpack suites for mac & linux, adding to unittest.py/factory.py	all test/builder masters and schedulers	med	landed	reconfig	lsblakk
in production	20101017 2030-2105 PDT	bug 605056	Lost unit tests in tinderbox mail notifiers	pm01:8010, pm03:8010, pm02:8011, buildbot-master{1,2}:8010	high	landed	reconfig	aki
in production	20101017 1930 PDT	bug 605056	Lost unit tests in tinderbox mail notifiers	all test masters	high	landed	reconfig	aki
in production	20101017 1510ish PDT	bug 579566	Add android to tm01's platform list	tm01	medium	landed	restart	bear
in production	20101017 1300-1600 PDT	bug 596852	purge_builds should delete directories older than 28 days	*	medium	landed	restart	nthomas
		bug 601807	Reset schedulerdb password			N/A
		bug 603133	1.9.1 tests broken unpacking tests			landed
		bug 579566	Create factory/steps to establish remote-testing environment for tegra			landed: bbc bb-c
in production	20101014 2021 PDT	bug 604566	Fix upload for try android	pm02:8011	medium	landed	reconfig	nthomas
in production/backed out	20101012 1600 PDT	bug 602634 -- landed; bug 603822 -- backed out	Remove android from s-c; attempt to fix Android nightly signing (backed out)	scheduler_master, 08x builders	low/high	lande/backed out	reconfig	lsblakk, aki
in production	20101012 0730 PDT	bug 602263, bug 602290, bug 595237, bug 505512, bug 602640, bug 602180, bug 601694, bug 601117	clean production slave lists, disable d2d/d3d/opengl tests, fix purple status, disable https poller, stop throwing exceptions in unpacktest, unittest timeout -> orange, update minidump_stackwalk	all scheduler, 08x builder, test masters	medium	landed	reconfig	lsblakk, armenzg, bhearsum, ssalbiz, ted
in production	20101008 1900 PDT	-	Cedar not building	scheduler master	low	-	reconfig (w/out update of configs/buildbotcustom)	aki
in production	20101005 1830ish PDT	bug 595977, bug 601323	Fennec 4.0b1 build2, release desktop fennec upload	pm02:8010	high	landed	reconfig	aki
in production	20101005 1340 PDT	bug 602037	MajorUpdateFactory should not be running buildN snippet script	release masters (pm:8010/8011, pm02:8010)	high	landed	reconfig	bhearsum
in production	20101005 0630 PDT	bug 600608	bustage fix - remove .hgrc from Linux slaves	all linux slaves	high	landed	N/A	armenzg
in production	20101005 0700-ish PDT	bug 600073	turn off perf testing on e10s branch	tm*/bm* test masters	medium	landed	reconfig
in production	20101005 0700-ish PDT	bug 600029	stop running ts cold on windows	tm*/bm* test masters	medium	landed	reconfig
in production	20101005 0700-ish PDT	bug 505512	turn infrastructure errors purple	all 0.8.0 masters	medium	landed	reconfig
in production	20101005 0700-ish PDT	bug 556530	move talos into Mercurial	all test masters	medium	landed	reconfig
in production	20101005 0700-ish PDT	bug 594018	selectively unpack test suites for unittests	all 0.8.0 masters	medium	landed	reconfig
in production	20101005 0700-ish PDT	bug 600626	use the last try: in a push's commit	0.8.0 scheduler masters	medium	landed	reconfig
in production	20101005 0700-ish PDT	bug 506404	add support for warnings on certain exit codes to ScriptFactory/fix exit codes in nanojit script	all 0.8.0 masters	medium	landed	reconfig
backed out	20101005 0700-ish PDT	bug 596852	purge_builds should delete directories older than some threshold	all masters	medium	landed	reconfig
in production	20100928 1130 PDT	bug 506404	bustage fix after nanojit patches (fixes shared reference issue	pm01,03, bm01,02	high	landed	reconfig	bhearsum/catlee
in production	20100928 0900-ish PDT	bug 588950	new buildbot masters in santa clara	buildbot-master1/2, all instances	high	landed	n/a (new masters)	bhearsum
in production	20100927 12:04 PDT	bug 598358	sync up the talos suites available to disposable branches with those of m-c	test-masters* and test-scheduler	low	landed	reconfig	lsblakk
in production	20100927 12:04 PDT	bug 585286	deactivate screensaver on fedora	test-m01,02, talos-m02	low	landed	reconfig	jhford
in production	20100926 1945 PDT	bug 571367	Leopard tests on new universals	talos-master02 (tm02 by mistake)	high (beta7 blocker)	landed	reconfig	nthomas
in production	20100926 1600 PDT	bug 571367	Mac universals become i386+x86_64	pm01,pm02:8011,pm03 + schedulers; tm01, tm02, talos-master02	high (beta7 blocker)	landed: 1 2 3 4 5 6	restart (masters were down for db outage)	nthomas
in production	20100920 12:36 PDT	bug 554249	fix linux m-c fennec upload location; enable dep buildsymbols	pm02	low	landed	reconfig	aki
in production	20100920 08:21 PDT	bug 327076	use perl script for unpacking complete mars	pm01,pm03	low	landed	reconfig	coop
in production	20100917 11:09 PDT	bug 597432	fix the repo url in misc_scheduler.py	test-scheduler, scheduler-master	low	landed	reconfig	lsblakk
in production	20100916 17:20 PDT	bug 593172, bug 527794	disable m-c, project branch maemo4 builds; mobile desktop make upload	pm01,pm02,pm03,scheduler,try	low	landed	reconfig	aki
in production	20100916 17:20 PDT	bug 596944	mobile l10n backlog	pm02	high	landed	reconfig	aki
in production	20100916 15:29 PDT	bug 591563	try server scheduler refactoring to look at sendchange comments first, then pushlog	scheduler, test-scheduler	low	landed	reconfig	lsblakk
in production	20100915 1916 PDT	bug 554249	android nightly updates	pm01, try, pm03, scheduler	high	landed	reconfig	aki
in production	20100914 1140 PDT	bug 594853	tracking bug for 3.6.10	pm:8011	high	landed	reconfig	bhearsum
in production	20100914 0745 PDT	bug 596223	set-up 3rd master capable of doing releases	new master on pm	high	landed	n/a	bhearsum
in production	20100913 18:05 PDT	bug 595730,bug 591690,bug 595232	fix win32 debug unittest on try regression, improve syntax, add opengl suite	scheduler_master (at 18:05 PDT), tests-scheduler (at 21:35 PDT)	med	landed	reconfig	lsblakk
in production	20100913 07:30 PDT	bug 497041	FtpPoller unchanged by buildbot reconfig	pm01,02	low	landed	reconfig	bhearsum
in production	20100909	bug 586567	Disable "OK to launch" dialog on 10.6 slaves	puppet	med	landed	n/a	bear
in production	20100909 0715 PDT	bug 580410	Enable OpenGL test suite for Mac 10.5	test masters	med	landed	reconfigure	armenzg
in production	20100909 0715 PDT	bug 593852	Remove talos-r3-w7-040 from our configs	n/a	low	landed	n/a	armenzg
in production	20100909 0650 PDT	bug 399095	offer updates between buildX -> buildY builds	pm,pm02	low	landed	reconfig	bhearsum
in production	20100909 0650 PDT	bug 409449	automatically verify test channel contents match the corresponding live channel contents	pm,pm02	low	landed	reconfig	bhearsum
in production	20100902 1030 PDT	bug 589814	deploy ANGLE libraries	Win7 testing minis	medium	deployed	N/A	armenzg
in production	20100902 0645 PDT	bug 505520	Run fuzzing jobs during idle time	pm*	medium	landed	reconfig	catlee
in production	20100902 0645 PDT	bug 589765	l10n changes are being merged	pm*	medium	landed	reconfig	catlee
in production	20100902 0645 PDT	bug 590011	Change buildbot db password	pm*, tm*	medium	landed	restart	catlee
in production	20100902 0645 PDT	bug 589396	Disable tgfx	tm*	low	landed	reconfig	catlee
in production	20100902 0645 PDT	bug 463113	ssh batch mode	puppet-master	low	landed		catlee
in production	20100902 0645 PDT	bug 589167	Run ts only once per changeset	tm*	medium	landed	reconfig	catlee
in production	20100902 0645 PDT	bug 557336	MozillaTestFactory	all	medium	landed	reconfig	jhford
backed out	20100902 0645 PDT	bug 592704	buildbotcustom.steps.SendChangeStep needs to watch stdout for success	pm*	low	landed	reconfig	bhearsum
in production	20100902 0645 PDT	bug 505512	Make infrastructure related problems turn the tree a color other than red	pm*, tm*	medium	landed	reconfig	bhearsum
in production	20100901 1429 PDT	bug 587315	Increase build space on macosx	pm*	low	landed	reconfig	buildduty (lsblakk)
in production	20100901 0730 PDT	bug 590197	Manage dock and screen resolution of Mac testing slaves throuhg puppet	n/a	med	ready to land	n/a	armenzg

Older Changes