ReleaseEngineering/Maintenance

From MozillaWiki
Jump to navigation Jump to search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

Are you changing the tool chain on a master? If so, let auto-tools know so they can update their masters

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems.

Did you change the tool chain on a master? If so, let auto-tools know so they can update their masters

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Outcome When Bug #(s) - Description(s)
in production 2012-05-22 1700 PDT
  • bug 754291 - buildbot master Makefile refers to the wrong hg
  • bug 756463 - Please reset the priority of the Oak branch
  • bug 753501 - followup, make try look for the Android XUL tooltool manifest in the path where it actually is
in production 2012-05-18 1313 PDT
  • bug 756463 - Please bump the priority of the Oak branch temporarily
  • bug 753132 - Fix env for nightly pgo builds
  • bug 573722 - set IS_NIGHTLY env var for nightly builds
  • bug 753132 - Do periodic PGO builds on pgo_platform too
in production 2012-05-18 0815 PDT
  • bug 756463 - Please bump the priority of the Oak branch temporarily
in production 2012-05-17 0750 PDT
  • bug 755434 - l10n repacks should not execute config.py
  • bug 753132 - Do 32-bit PGO builds on win64
  • bug 751158 - Create tcheckboard3 to measure checkerboard with low res off
  • bug 755989 - setup-master.py doesn't set up staging symlinks properly
  • bug 743304 - After SSH failure, Android XUL mozilla-central nightly builder spams a bunch of "SyntaxError: invalid syntax" when running retry.py / balrog-client.py
  • bug 753501 - Add empty tooltool manifests to some platforms
in production 2012-05-14 1118 PDT
  • bug 753488 - Android native on Aurora -> multilocale.
in production 2012-05-14 0830 PDT
  • bug 750837 - 13.0b2 build 2 configs. r=hwine
  • bug 754517 - disable larch and enable pine rentable branches, r=bear
  • bug 747500 - setup-master.py refers to files which have been removed. r=catlee
  • bug 753132 - Use win64 machines for 32-bit pgo builds on build-system branch. r=rail
  • bug 754373 - Use firefox-tuxedo.ini for Thunderbird builds. r=standard8
  • bug 754397 - Disable signing at build time for Thunderbird. r=nthomas
  • bug 754430 - Missing mozilla/ dir in Thunderbird beta build. p=standard8,r=jhopkins
  • bug 701783 - remove scratchbox references from buildbotcustom. r=catlee
  • bug 753132 - Support 'pgo_platform' key for deciding which machines do PGO builds. r=rail
  • bug 750744 - Test and deploy SUT agent 1.08. r=bear
in production 2012-05-11 0930 PDT
  • bug 754297 - add sys.stdout.flush() to sut_tools' scripts
in production 2012-05-11 0800 PDT
  • bug 746260 - disable the screen resolution changing on android for jsreftest and crashtest, leave it on for reftest.
  • bug 753868 - use aus3-staging for Thunderbird release builds, r=jhopkins
  • bug 744601 - tracking bug for build and release of Thunderbird 13.0b2. r=standard8
  • bug 752531 - migrate dev-stage01 to scl3. r=rail
in production 2012-05-10 1801 PDT
  • bug 753488 - make FN multi on m-c only, reenable nightly updates.
  • bug 753625 - Move all Thunderbird branches onto Firefox infra
  • bug 749748 - kill l10n verify.
  • bug 753868 - Use aus3-staging.mozilla.org for Thunderbird release builds.
  • bug 753865 - Email thunderbird-drivers for Thunderbird release builds.
  • bug 748157 - Load thunderbird_release_branches from master_config.json
in production 2012-05-09 0800 PDT
  • bug 752373 - Stop running Android crashtest-1 until someone's ready to fix it
  • bug 751070 - retire sjc1 VMs
  • bug 750031 - moz2-darwin10-slave02 problem tracking
  • bug 746201 - Remove unresolved machines from buildbot-configs
  • bug 752430 - Swap comm-aurora over to Firefox infra
  • bug 749051 - TryChooser: could -n be the default?
  • bug 751878 - OSError: [Errno 13] Permission denied: '/home/ftp' for pvtbuilds2.dmz.scl3.mozilla.com
in production 2012-05-03 1325 PDT
backed out 2012-05-03 1200 PDT
  • Backout 4ab5af03cce1 (new scl3 slaves). r=backout
in production 2012-05-03 1000 PDT
  • bug 751165 - revert higher priority for m-i. r=philor,ehsan
  • bug 744067 - new scl3 slaves; r=coop
  • bug 744067 - new scl3 slaves (must be in staging); r=aki
  • Add ACTIVE_THUNDERBIRD_RELEASE_BRANCHES. r=armenzg
  • bug 751895 - Preproduction release master fails trying to checkconfig. r=jhopkins
  • bug 750973 - copy in-tree m-a linux32 mozconfig into mozilla2 to fix aurora source release. r=catlee
in production 2012-05-03 08:00 PDT
  • bug 751506 - No 10.7 32-bit debug builders on Thunderbird trees. r=coop
  • bug 748628 - Switch Thunderbird builds to use OS X 10.7 build machines. Add in the 'TB ' prefix to match the other Thunderbird builders. r=jhopkins
  • bug 744864 - Update list of l10n modules that trigger changes. r=Pike
  • bug 751560 - Temporarily disable uploading symbols on Windows 32 bit try-comm-central builds. r=jhopkins
  • bug 751514 - Thunderbird bloat test builders should warn and halt on failure, not error on failure. r=jhopkins
in production 2012-05-02 12:00 PDT
  • bug 750635 - Swap try-comm-central over to pushing to the thunderbird product directory, and get it running unit tests.
  • Follow-up to bug 748628, fix some more issues with the Thunderbird lion builders - the names and the ccache settings.
  • bug 739994 - Remove references to 10.5 platform and associated slaves in configs - r=jhford
in production 2012-05-02 08:00 PDT
  • bug 748628 - Switch Thunderbird builds to use OS X 10.7 build machines. r=coop
  • bug 743304 - After SSH failure, Android XUL mozilla-central nightly builder spams a bunch of "SyntaxError: invalid syntax" when running retry.py / balrog-client.py. r=catlee
  • bug 751165 - Bump priority of mozilla-inbound to help open the tree earlier. r=catlee
in production 2012-05-01 19:00 PDT
in production 2012-04-30 13:30 PDT
  • bug 750305 - Use comm-central as reference branch for try-comm-central builds
  • bug 749596 - Enable aurora nightly updates (April 27th, 2012 edition)
in production 2012-04-30 11:30 PDT
  • bug 749867 - Don't try to build SpiderMonkey --enable-shark builds on 10.7 where there is no Shark, r=coop
  • buildbot-configs patch to reflect new all-locales locations (Bug 711534 - Configure Thunderbird release builders) r=standard8
  • bug 669428 - Run Jetpack tests on mozilla-inbound, r=armenzg
  • bug 748633 - Thunderbird try logs failing to upload. r=rail
in production 2012-04-27 11:30 PDT
  • bug 749524 - Upload comm-aurora snippets to comm-aurora-test channel
  • bug 711534 - Configure Thunderbird release builders
  • bug 749288 - linux comm-central builds use wrong python when calling balrog client
  • bug 749494 - Re-enable graph server for staging/preproduction
  • bug 729392 - Install toolchain needed for SPDY testing onto test machines
  • bug 745300 - Do Mac spidermonkey builds on 10.7
in production 2012-04-26 11:00 PDT
  • bug 749076 - tooltool should be invoked with -o (--overwrite) option
  • bug 739802 - disable b2g on aurora, beta, release
backed-out 2012-04-26 09:00 PDT
in production 2012-04-26 07:00 PDT
in production 2012-04-25 22:26 PDT
  • bug 742131 - fix upload host for windows try symbols
in production 2012-04-25 12:00 PDT
  • bug 743977 - turn off balrog client for staging and preproduction builds
  • bug 723340 - move dm-pvtbuild01 to a new datacenter
  • bug 747821 - Need to run tpr_responsiveness on Try until it's not run anywhere anymore
  • bug 729667 - re-create the services on dm-wwwbuild01 in scl3
in production 2012-04-24 6:30 PDT
  • bug 747966 - comm-central builds not firing automatically
  • bug 747862 - Disable shark nightly builds on Thunderbird builders
in production 2012-04-23 15:30 PDT
  • bug 746708 - Updates builder fails running backupsnip and pushsnip
  • bug 747756 - Bump "'make hg-bundle" timeout
  • bug 747892 - mozilla-release's releasetestUptake value should be set to 1
  • bug 747460 - consolidate windows peptest config files
in production 2012-04-18 0645 PDT
  • bug 745545 - Handle Thunderbird revisions in NightlyRepackFactory.
  • bug 745547 - Move talosCmd into SUITES loop (generateTalosBranchObjects).
  • bug 745299 - Add hg-internal as a mirror.
  • bug 745500 - Turn on robocop testCheck2 on tinderbox builds.
  • bug 735390 - 12.0b6 configs + fix test-masters.sh + move l10n-changesets_mobile-aurora.json into mozilla/.
  • bug 746537 - Increase priority for Birch, drop Maple back down
in production 2012-04-17 1211 PDT
  • bug 746159 - make birch be like inbound
  • bug 739994 - turn off spidermonkey builds on 10.5
  • bug 744098 - switch xulrunner osx builds to upload tarballs
  • bug 732976 - singlesourcefactory should generate checksums
in production 2012-04-17 0630 PDT
in production 2012-04-12 1830-45 PDT
in production 2012-04-10 various times
in production 2012-04-10 1100 PST
  • bug 744049 - tcheckerboard always reports 1.0 (tegra talos web server updated to talos tip)
in production 2012-04-09 0700 PDT
  • bug 607392 - split tagging into en-US and other
  • bug 721885 - shut off unused branch
  • bug 400296 - Have release automation support signing OSX builds (up to 10.7 support)
in production 2012-04-04 11:00 PDT
  • bug 690311 - deploy newer version of cleanup.py to the foopies
in production 2012-03-30 6:15 PDT
  • bug 738588 - add ts_paint to the android tests.
  • bug 737458 - replace tpr_responsiveness for tp5row.
  • bug 737458 - add tpr_responsiveness temporarily for mozilla-central and larch.
  • bug 740599 - update staging release config files;
in production 2012-03-29 6:55 PDT
  • bug 715193 - If a branch does not use talos_from_source_code we should fallback to talos.mobile.old.zip (fixes esr10).
in production 2012-03-28 16:35 PDT
  • bug 740196 - ts_paint on Android doesn't actually work
in production 2012-03-28 11:55 PDT
  • bug 737632 - Remove jaegermonkey, graphics and pine to reduce builders
  • bug 723667 - fix Android trobocheck and ts_paint tests.
  • bug 739486 - test-masters.sh should run ./setup_master.py -t
  • add option to setup_master.py to print error logs when hit
  • bug 723667 - enable trobopan and tcheckerboard by default (not for m-a/m-b/m-r/1.9.2)
  • bug 627182 - Automate signing and publishing of XULRunner builds. r=bhearsum
in production 2012-03-27 12:30 PDT
  • bug 723667 - Add trobopan and trobocheck to m-c/m-i. r=jmaher
in production 2012-03-23 11:30 PDT
in production 2012-03-27 01:35 PDT
in production 2012-03-23 7:00 PDT
  • bug 737864 - Tweak release category for Thunderbird.
  • bug 737458 - add tp5row side by side and cleanup config.py.
  • bug 737581 - enable peptest on m-c and m-i.
  • bug 713846 - Treat 'fennec' builds as having product 'mobile' for the purposes of uploading logs.
backout 2012-03-21 11:45 PDT
  • bug 737427. Use 1024x768 as the screen resolution for the tegras.
in production 2012-03-21 9:30 PDT
  • bug 737427. Use 1024x768 as the screen resolution for the tegras.
  • bug 697150. (Bv1) Remove 'ac_add_options --disable-installer' for XulRunner current branches.
  • bug 733394. Add leak test logic to mozilla-beta.
  • bug 736587. Enable Android for pine.
in production 2012-03-20 8:30 PDT
in production 2012-03-19 10:30 PDT
in production 2012-03-16 8:45 PDT
  • bug 734996 - fennec beta release update channel -> beta.
  • bug 734221 - deploy updateSUT.py and upgrade the boards to SUT Agent version 1.07.
  • bug 734996 - source: get a nonce earlier
in production 2012-03-13 17:00 PDT
  • bug 735201 - Remove leading ../ from symbols path for tegras
  • bug 735421 - Disable Aurora updates until the Aurora 13 has stabilized
in production 2012-03-12 16:00 PDT
  • bug 734417 - enable mobile builds on the profiling branch
  • bug 731617 - No nightly builds on maple branch since 27 Feb
  • bug 732285 - Set MINIDUMP_STACKWALK for Android
  • bug 733668 - Include "ERROR: We tried to download the talos.json file but something failed" and "ERROR 500: Internal Server Error" for Talos hgweb operations to RETRY
  • bug 630518 - l10n verify, update verify, and final verification builders need to set "branch" when reporting to clobberer
in production 2012-03-08 09:00 PT
  • bug 731814 - Add checks that we're not exceeding max # of builders per slave.
  • bug 731617 - Remove win64 for now in maple.
  • bug 731339 - Remove slaves that are not production
  • bug 732730 - Remove non-functional and unwanted pgo_platforms overrides
  • bug 732110 - remove buildbot-configs/mozilla2/mobile
  • bug 728271 - Post to graphs.m.o instead of graphs-old.m.o
  • bug 729144 - Post to graphs.allizom.org.
  • bug 723667 - Add robocop disabled.
  • bug 730050 - TryBuildFactory looks in the wrong place for malloc.log
  • bug 712538 - leaktest parity on try
  • bug 723667 - Use talos.zip for tegras and prep work for talos robocop
in production 2012-03-06 06:30 PT
  • bug 732500 - Enable nightly updates on maple
  • bug 732699 - ESR release automation should push to mirrors automatically
  • bug 730918 - Android on esr10 is busted, no doubt by branding since that always seems to be the problem
  • bug 561754 - Don't download symbols for test runs, pass symbol zip URL as symbols path
  • bug 732516 - l10n verification shouldn't rsync zip files
  • bug 732468 - Add the ridiculous "abort: error:" to the list of hg errors that trigger RETRY
in production 2012-03-01 7:30 PT
  • bug 721360 - Do what changeset 9a0c428bdb69 really wanted to do.
  • bug 561754 - Disable symbol download on demand for mozilla-1.9.2 branch.
  • bug 660480 - mark as RETRY for common tegra errors
  • bug 729918 - start_uptake_monitoring builder uses wrong script_repo_revision property.
  • bug 561754 - Download symbols on demand by default for desktop unittests.
in production 2012-02-27 7:45 PT
  • bug 729077 - recycle talos-r4-lion-083 and talos-r3-snow-081 as mac-signing[12]
  • Fix up staging and preproduction test slave lists.
  • bug 729426 - Do periodic PGO on services-central
  • bug 727580 - linux-android for esr10, without merging 11.0 to m-r.
in production 2012-02-21 9:30 PT
  • bug 719511 - add optional reboot command to ScriptFactory
  • bug 725292 - some repacks failed in 11.0b2 because of missing tokens
  • bug 728104 - AggregatingScheduler resets its state on reconfig
  • bug 722608 - Remove android signature verification
  • bug 719260 - Investigate why updates builder triggered twice for 10.0b5
  • bug 719511 - Reenable peptest + add reboot_command
  • bug 712678 - android-xul different update channel from android
in production 20120217 1148 PST
  • bug 721822 - remove talos_from_code.py from the tools repo
in production 20120214 1245 PST
  • bug 726901 - adjust resolution for reftests to 1600x1200
  • bug 689989 - restore /system/etc/hosts on testing tegras
in production 20120213 1200 PST
in production 20120208 01:20 PST
in production 20120207 13:00 PST
  • bug 719443 - add robocop unitttest testtype
  • bug 715715 - download & install robocop for robocop test suites
  • bug 725046 - Re-enable mobile aurora updates
  • bug 554324 - Only set MOZ_PKG_VERSION when appVersion != version
  • [BACKED OUT] - bug 719511 - optional ScriptFactory reboot().
in production 20120202 15:50 PST
  • bug 723743 - android native to en-US (no multilocale); disable android-xul single locale repacks.
  • bug 719697 - --disable-tests on android* l10n-mozconfigs.
  • bug 723277 - don't enable remot

Android Testing

Web Server Cluster

Revision Bug # When Who
488bc187a3ef bug 753822 20120510 1045 AM PDT armenzg

Find here instructions to see how to update this.

clientproxy servers

Production

Talos Rev Pageloader Rev Taras Bench Rev sut_tools Bug # When Who
2a995b4ed124 31249cbe4f19 bfc910cd8dd3 ae5d6911905a talos: bug 629503 20110202 23:00 PDT bear

Pending

Talos Rev Pageloader Rev Taras Bench Rev sut_tools Bug # When Who

Servers:

  • bm-foopy01.build.mozilla.org
  • bm-foopy02.build.mozilla.org
/builds/cp
/builds/talos-data/talos
/builds/talos-data/talos/pageloader@mozilla.org
/builds/talos-data/talos/bench@taras.glek
/builds/sut_tools
Retrieved from "https://wiki.allizom.org/index.php?title=ReleaseEngineering/Maintenance&oldid=433557"