This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

Are you changing the tool chain on a master? If so, let auto-tools know so they can update their masters

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems.

Did you change the tool chain on a master? If so, let auto-tools know so they can update their masters

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Outcome	When	Bug #(s) - Description(s)
in production	2012-04-09 0700 PDT	bug 607392 - split tagging into en-US and other bug 721885 - shut off unused branch bug 400296 - Have release automation support signing OSX builds (up to 10.7 support)
in production	2012-04-04 11:00 PDT	bug 690311 - deploy newer version of cleanup.py to the foopies
in production	2012-03-30 6:15 PDT	bug 738588 - add ts_paint to the android tests. bug 737458 - replace tpr_responsiveness for tp5row. bug 737458 - add tpr_responsiveness temporarily for mozilla-central and larch. bug 740599 - update staging release config files;
in production	2012-03-29 6:55 PDT	bug 715193 - If a branch does not use talos_from_source_code we should fallback to talos.mobile.old.zip (fixes esr10).
in production	2012-03-28 16:35 PDT	bug 740196 - ts_paint on Android doesn't actually work
in production	2012-03-28 11:55 PDT	bug 737632 - Remove jaegermonkey, graphics and pine to reduce builders bug 723667 - fix Android trobocheck and ts_paint tests. bug 739486 - test-masters.sh should run ./setup_master.py -t add option to setup_master.py to print error logs when hit bug 723667 - enable trobopan and tcheckerboard by default (not for m-a/m-b/m-r/1.9.2) bug 627182 - Automate signing and publishing of XULRunner builds. r=bhearsum
in production	2012-03-27 12:30 PDT	bug 723667 - Add trobopan and trobocheck to m-c/m-i. r=jmaher
in production	2012-03-23 11:30 PDT	bug 627182 bug 738685 bug 734223 bug 738286 bug 719491 bug 737656 bug 715193 bug 702595 bug 735383
in production	2012-03-27 01:35 PDT	bug 739505 - Fix talos on beta
in production	2012-03-23 7:00 PDT	bug 737864 - Tweak release category for Thunderbird. bug 737458 - add tp5row side by side and cleanup config.py. bug 737581 - enable peptest on m-c and m-i. bug 713846 - Treat 'fennec' builds as having product 'mobile' for the purposes of uploading logs.
backout	2012-03-21 11:45 PDT	bug 737427. Use 1024x768 as the screen resolution for the tegras.
in production	2012-03-21 9:30 PDT	bug 737427. Use 1024x768 as the screen resolution for the tegras. bug 697150. (Bv1) Remove 'ac_add_options --disable-installer' for XulRunner current branches. bug 733394. Add leak test logic to mozilla-beta. bug 736587. Enable Android for pine.
in production	2012-03-20 8:30 PDT	bug 723667 - enable talos robocop for pine and only for native tests bug 737077 - re-enable aurora updates bug 713846 - unified log handling bug 734320 - fix jetpack log parsing bug 737049 - run reftest-no-accel correctly bug 723386 - fix reserved slaves handling
in production	2012-03-19 10:30 PDT	bug 736284 - re-enable aurora updates
in production	2012-03-16 8:45 PDT	bug 734996 - fennec beta release update channel -> beta. bug 734221 - deploy updateSUT.py and upgrade the boards to SUT Agent version 1.07. bug 734996 - source: get a nonce earlier
in production	2012-03-13 17:00 PDT	bug 735201 - Remove leading ../ from symbols path for tegras bug 735421 - Disable Aurora updates until the Aurora 13 has stabilized
in production	2012-03-12 16:00 PDT	bug 734417 - enable mobile builds on the profiling branch bug 731617 - No nightly builds on maple branch since 27 Feb bug 732285 - Set MINIDUMP_STACKWALK for Android bug 733668 - Include "ERROR: We tried to download the talos.json file but something failed" and "ERROR 500: Internal Server Error" for Talos hgweb operations to RETRY bug 630518 - l10n verify, update verify, and final verification builders need to set "branch" when reporting to clobberer
in production	2012-03-08 09:00 PT	bug 731814 - Add checks that we're not exceeding max # of builders per slave. bug 731617 - Remove win64 for now in maple. bug 731339 - Remove slaves that are not production bug 732730 - Remove non-functional and unwanted pgo_platforms overrides bug 732110 - remove buildbot-configs/mozilla2/mobile bug 728271 - Post to graphs.m.o instead of graphs-old.m.o bug 729144 - Post to graphs.allizom.org. bug 723667 - Add robocop disabled. bug 730050 - TryBuildFactory looks in the wrong place for malloc.log bug 712538 - leaktest parity on try bug 723667 - Use talos.zip for tegras and prep work for talos robocop
in production	2012-03-06 06:30 PT	bug 732500 - Enable nightly updates on maple bug 732699 - ESR release automation should push to mirrors automatically bug 730918 - Android on esr10 is busted, no doubt by branding since that always seems to be the problem bug 561754 - Don't download symbols for test runs, pass symbol zip URL as symbols path bug 732516 - l10n verification shouldn't rsync zip files bug 732468 - Add the ridiculous "abort: error:" to the list of hg errors that trigger RETRY
in production	2012-03-01 7:30 PT	bug 721360 - Do what changeset 9a0c428bdb69 really wanted to do. bug 561754 - Disable symbol download on demand for mozilla-1.9.2 branch. bug 660480 - mark as RETRY for common tegra errors bug 729918 - start_uptake_monitoring builder uses wrong script_repo_revision property. bug 561754 - Download symbols on demand by default for desktop unittests.
in production	2012-02-27 7:45 PT	bug 729077 - recycle talos-r4-lion-083 and talos-r3-snow-081 as mac-signing[12] Fix up staging and preproduction test slave lists. bug 729426 - Do periodic PGO on services-central bug 727580 - linux-android for esr10, without merging 11.0 to m-r.
in production	2012-02-21 9:30 PT	bug 719511 - add optional reboot command to ScriptFactory bug 725292 - some repacks failed in 11.0b2 because of missing tokens bug 728104 - AggregatingScheduler resets its state on reconfig bug 722608 - Remove android signature verification bug 719260 - Investigate why updates builder triggered twice for 10.0b5 bug 719511 - Reenable peptest + add reboot_command bug 712678 - android-xul different update channel from android
in production	20120217 1148 PST	bug 721822 - remove talos_from_code.py from the tools repo
in production	20120214 1245 PST	bug 726901 - adjust resolution for reftests to 1600x1200 bug 689989 - restore /system/etc/hosts on testing tegras
in production	20120213 1200 PST	bug 725727 - reduce # of chunks for update_verify. bug 607392 - split tagging into en-US and other.
in production	20120208 01:20 PST	bug 723954 - 11.0b2 configs bug 718385 - android single locale updates bug 717106 - Release automation for ESR
in production	20120207 13:00 PST	bug 719443 - add robocop unitttest testtype bug 715715 - download & install robocop for robocop test suites bug 725046 - Re-enable mobile aurora updates bug 554324 - Only set MOZ_PKG_VERSION when appVersion != version [BACKED OUT] - ~~bug 719511 - optional ScriptFactory reboot().~~
in production	20120202 15:50 PST	bug 723743 - android native to en-US (no multilocale); disable android-xul single locale repacks. bug 719697 - --disable-tests on android* l10n-mozconfigs. bug 723277 - don't enable remot

Android Testing

Web Server Cluster

Revision	Bug #	When	Who
3efbac1f685a	unknown	unknown	unknown

Update Procedure:

ssh to bm-remote-talos-webhost-01
cd /var/www/html/talos
hg pull && hg up
rsync -azf --delete . bm-remote-talos-webhost-02:/var/www/html/.
rsync -azf --delete . bm-remote-talos-webhost-02:/var/www/html/.

Servers:

bm-remote-talos-webhost-01.build.mozilla.org
bm-remote-talos-webhost-02.build.mozilla.org
bm-remote-talos-webhost-03.build.mozilla.org

clientproxy servers

Production

Talos Rev	Pageloader Rev	Taras Bench Rev	sut_tools	Bug #	When	Who
2a995b4ed124	31249cbe4f19	bfc910cd8dd3	ae5d6911905a	talos: bug 629503	20110202 23:00 PDT	bear

Pending

Talos Rev	Pageloader Rev	Taras Bench Rev	sut_tools	Bug #	When	Who