ReleaseEngineering/Maintenance

From MozillaWiki
Jump to navigation Jump to search

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

Are you changing the tool chain on a master? If so, let auto-tools know so they can update their masters

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems.

Did you change the tool chain on a master? If so, let auto-tools know so they can update their masters

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Outcome When Bug #(s) - Description(s)
in production 20120131 2100 PST
  • bug 718777 - updating configs for mozilla-beta so we can get native builds going
  • bug 722719 - change to SDK 14
  • bug 722951 - (temporarily) redirect aurora updates to test channel
  • bug 722940 - codesize upload broken for SeaMonkey [and Thunderbird] due to tools dir being incorrect
  • bug 718777 - Tracking bug for build and release of Firefox/Fennec 11.0b1. Poll signed Fennec APKs for all signed
  • bug 708656 - Use signing on demand for releases. Use AggregatingScheduler for repack_complete
  • bug 719260 - Investigate why updates builder triggered twice for 10.0b5
  • bug 660480 - RETRY on common tegra errors
in production 20120127 1130 PST
  • bug 719697 - robocop isn't signed properly from buildbot builds
in production 20120127 1040 PST
in production 20120127 0730 PST
  • bug 719544. talos_from_source.py - Make the pine branch to allow downloading talos.zip from any place like on 'try'
  • bug 717662 - Please disable debug builds and tests on the profiling branch
  • bug 720782 - If we dont_build a platform on project_branches we should not add testers for it
  • bug 721360 - Bug 698827 - Run 10.5 leak builds on 10.6 machines for aurora
  • bug 721573 - Sign the profile branch nightlies using the m-c nightly key
  • bug 717106 - Release automation for ESR
  • bug 698827 - Bug 698827 - Run 10.5 leak builds on 10.6 machines for aurora
  • bug 715966 - branch 1.9.2 confusingly set on talos tbpl logs
  • bug 718828 - Don't wait for NFS cache at the end of the updates builder
  • bug 705403 - Sendchanges [on windows] from build steps are being done from old buildbot version - backed out
  • bug 683417 - retry.py didn't actually kill process tree for a timed-out pushsnip
  • bug 673834 - Obsolete ReleaseRepackFactory, fold logic into CCReleaseRepackFactory
in production 20120123 1435 PST
  • bug 719859 - remove double posting ts_paint and tpaint. p=armenzg
  • bug 718445 - stage-old should be referenced as stage in scripts/configs. p=bhearsum
in production 20120123 1405 PST
  • bug 649641 - use ntpd on linux32/linux64 ix slaves
in production 20120123 1140 PST
  • bug 711619 - Add Android builds+tests and periodic PGO on the Fx-Team branch, p=philor
  • bug 719859 - Side by side on mozilla-central for ignore_first changes. p=jmaher
in production 20120123 0730 PST
  • bug 705403 - Sendchanges [on windows] from build steps are being done from old buildbot version
  • bug 719772 - Sign Callek up for the full release process e-mails
  • bug 716561 - reevaluate which release mail gets sent to release-drivers
  • bug 561198 - compress leak test / codesighs logs prior to uploading
  • bug 699219 - Add automated clean up of hg-shared directory
  • bug 714284 - L10n mac dep builds busted on central and aurora
  • bug 719261 - Add more logging to AggregatingScheduler
in production 20120119 1200 PST
in production 20120119 1100 PST
in production 20120117 1230 PST
  • bug 695351 - android mochitests to use in-tree manifest
  • bug 700415 - peptest on try
  • bug 712750 - print more data for screenresolution in buildbot factories
in production 20120117 0800 PST
  • bug 698827 - Run 10.5 leak builds on 10.6 machines for try. p=armenzg
in production 20120116 1325 PST
  • Require branch parameter to clobberer HTML interface
in production 20120113 07:00 PST
  • bug 714490 - make hgtool handle mirror/master hg outages better
in production 20120112 16:40 PST
  • bug 712422 - add a --bootstrap cli flag to reftest/crashtest/jsreftest for android
  • bug 698425 - enable android and android-xul l10n repacks
  • Bustage fix. Changeset fa1c76238b7c
  • bug 713442 - point 1.9.2 release configs to the compare-locales RELEASE_0_8_2 tag
  • bug 717621 - Remove decomissioned slaves
  • bug 698425 - android and android-xul l10n mozconfig
  • bug 567274 - Talos should halt on download or unzip failure
in production 20120109 1806 PST
  • stage rather than masters
  • bug 712008 - Always trim revision to 12 chars
  • bug 716431 - Block asc files for partial mars in latest-<branch> dirs (stage
in production 20120106 1300 PST
  • bug 715623 - add --cachedir support to signtool.py
in production 20120104 1315 PDT
  • Back out 7a7847f7fc05 (bug 711275: Make sure appVersion changes with every Firefox 10 beta)
  • bug 712008 - Pass platform to post_upload.py for shark
  • bug 681948 - Automatically retry after a devicemanager.DMError
  • bug 715119 - [signing-server] Bump token TTL
  • bug 713161 - new high tegra added
  • bug 711221 - turn on create_snippet and create_partial for profiling branch
  • bug 712150 - bustage fix for linux,m-r and xulrunner in-tree mozconfig path
in production 20111222 0800 PDT
  • bug 710350 - Don't hard-code 'firefox' and 'fennec' in misc.py.
  • bug 707152 - enable leaktest for 10.6 everywhere except some release branches.
  • bug 711367 - enable android-xul tests
  • bug 673131 - Enable talos_from_source_code.
  • bug 712094 - re-enable aurora updates.
  • bug 711275 - Make sure appVersion changes with every Firefox 10 beta. r=rail
in production 20111221
  • bug 683734 - added a bunch of talos-r3 slaves to production
in production 20111221 1300 PST
in production 20111221 1100 PST
in production 20111220 0610 PST
  • bug 673131 - when minor talos changes land, the a-team should be able to deploy with minimal releng time required - backed-out
  • bug 704582 - [tracking bug] deploy 83 tegras
  • bug 712115 - L10n mac nightlies busted on central and aurora
  • bug 710453 - Release Engineering changes for the Firefox 11 merge to Aurora on Dec 20
  • bug 712094 - push mozilla-aurora updates to auroratest channel until merge stabilizes
  • bug 712068 - Adjust default releasetestUptake value
in production 20111219 1000 PST
in production 20111217 0800 PST
backed out 20111216 1700 PST
  • bug 705403 - Sendchanges [on windows] from build steps are being done from old buildbot version
in production 20111215 0830 PST
in production 20111214 0800 PST
  • bug 509158 Reduce default token time to 2 hours; fix last-complete-mar detection
  • bug 683734 Add new rev3 machines.
  • bug 708475 accept 'mochitest' and 'reftests' as synonyms for 'mochitests' and 'reftest' (with tests)
  • bug 708859 android signature verification should look for android-arm.apk
  • bug 709233 reenable android and android-xul multilocale for m-c nightlies
  • bug 709383 Turn off win64 signing on m-c
  • bug 709979 Set the branch property for projects/addon-sdk jobs to just addon-sdk
  • bug 710048 decrease interval between mozilla-inbound pgo builds
  • bug 710050 never merge pgo builds
  • bug 710085 Pass mozillaDir argument to NightlyBuildFactory
  • bug 710221 Implement AggregatingScheduler
in production 20111208 0920 PST
  • bug 509158 Fix nightly snippet generation, reduce default token time, and enable signing on inbound
  • bug 707666 Enable win64 signing on elm
  • bug 708341 Turn off android-xul talos tests
in production 20111206 1300 PST ish
  • bug 509158 Don't enable signing for l10n check steps.
  • bug 509158 Sign builds as part of the build process: enable signing server for debug builds; disable pre-signed updater on elm.
  • bug 671450 Try different sources for revision in log_uploader
  • bug 706832 Implement master side token generation for signing on demand.
  • bug 509158 Enable signing for mozilla-central windows builds.
  • bug 704549 reenable android native on m-c.
  • bug 703772 disable android-xul updates + uploadsymbols.
in production 20111205 0800 PST
  • bug 509158 - signing builds (elm/oak only, hopefully)
  • bug 706832 - Implement master side token generation for signing on demand. r=catlee,bhearsum
  • bug 671450 - Try different sources for revision in log_uploader - r=nthomas
  • bug 707152 - enable leaktests for m-i, try and m-c on macos64-debug. r=rail.
  • bug 706720 - Post to graphs-old. r=catlee
in production 20111202 1600 PST
in production 20111201 1100 PST
  • bug 694332 - Use make tier_nspr when building for l10n - r=armenzg
  • bug 693352 r=aki add minidump_stackwalk and symbols to the android automation
  • bug 705936 - reconfigs should re-generate master_config.json a=aki
in production 20111201 0900 PST
  • bug 704555 - deploy rss for tp4m on android (required android talos update)
in production 20111128 1448 PST
  • bug 701684 - remove mozilla-1.9.1 from config.py. r=bhearsum
  • add r4 slaves 080-085 to configs r=catlee
  • bug 705040 - reenable native android builds on try. r=bhearsum
  • bug 691483 - update MU to 3.6.24 -> 8.0.1, r=lsblakk
in production 20111124 0815 PST
in production 20111121 1300 PST
  • bug 702351 - enable tp_responsiveness on m-c
  • bug 700705 - remove more slaves
  • add talos-r4-snow-060 to 080 back to the pool
  • bug 692692 - re-enable PGO for Win64
  • bug 701766 - Remove tegra slaves that had not taken any jobs and are not coming back to production any time soon
  • bug 704200 - android dep builds permared after bug 701864; sometimes causing nightlies not to trigger - disable native android builders everywhere except birch
in production 20111118 0700 PST
  • bug 700513 - set BINSCOPE for win32 on try
  • bug 702631 - linux, linux64 and mac partner repacks aren't triggered
  • bug 703280 - Use dev-stage01 as SYMBOL_SERVER_HOST for staging try builds
  • bug 702351 - deploy talos.zip which includes responsiveness
in production 20111117 0600 PST
  • bug 702834 - Pass mozillaDir to dep factory.
  • bug 701864 - support mobile builds+repacks out of mobile/, mobile/xul/, and mobile/android/.
  • bug 701766 - remove staging tegras.
  • bug 700513 - Add BINSCOPE env var to win32, win32-debug, and win32-mobile
  • bug 701476 - split android reftests from 2 chunks to 3 chunks.
  • bug 702357 - enable new tegras for production
  • bug 702368 - add hangmonitor.timeout=0 pref to dirty jobs.
  • bug 702645 - win32_repack_beta broken due to "LINK : fatal error LNK1104: cannot open file 'mozcrt.lib'".
  • bug 548551 - Turn off arm nanojit builds.
  • bug 700705 - Remove a bunch of decomissioned slaves.
  • bug 683734 - remove talos-r3-snow machines, remove snowleopard-r4 platform, move talos-r4-snow to snowleopard platform
in production 20111116 0700 PST
  • bug 702351 - deploy talos.zip which includes responsiveness
in production 20111111 1712 PST
  • bug 697389 - multilocale birch android nightlies, against l10n-central.
  • bug 697404 - disable tp4m for birch
in production 20111110 1200 PST
  • bug 700901 - reorder mozconfig to get past mozconfig diff. p=aki
  • bug 700901 - fix l10n relbranch. p=aki
  • bug 701116 - Mobile desktop builds should be nightly-only. p=rail
  • bug 701113 - maemo tier 3 (removing all maemo references except mobile/) p=aki
  • bug 672132 - Run beta and release releases in preproduction. p=rail
  • bug 698946 - further setup-masters.py improvements p=jhford
in production 20111108 1630 PST
  • bug 699407 - Set mirror / bundle URLs. p=catlee
  • bug 700721 - update buildbot-configs for merge of nightly->aurora and aurora->beta p=lsblakk
  • bug 700453 - make test-master01 tegra specific. p=aki
  • bug 700794 - Disable aurora daily updates until merge to mozilla-aurora is good. p=armenzg
  • bug 700737 - Remove slaves given to Thunderbird. p=armenzg
in production 20111108 1100 PST bug 687064 - hgtool work. p=catlee
in production 20111107 0930 PDT bug 660124 - remove "paint" set. p=armenzg
in production 20111107 0845 PDT
  • bug 692812 - add ability to have pgo strategies p=jhford
  • bug 693771 - add 10.7 test slaves to buildbot configs p=jhford
  • bug 698837 - use signed updater.exe for elm and oak branches. p=bhearsum
  • bug 695921 - removing duplicated entry for ftp_url on jetpack p=lsblakk
  • bug 698837 - use signed updater.exe for elm and oak project branches. p=bhearsum
  • bug 660124 - replace ts/twinopen for ts_paint/tpain and some cleanup. p=armenzg
  • bug 699802 - enable_leaktests for m-i and try. p=armenzg
in production 20111028 1205 PDT
  • bug 695707 - mozharness should be tagged automatically for 8.0+ releases
  • bug 695921 - test per checkin addons-sdk against opt & debug across mozilla-{beta,central,aurora,release} latest tinderbox builds
in production 20111025 1200 PDT
  • bug 681855 - Frequent Tegra "Cleanup Device exception" or "Configure Device exception" from "Remote Device Error: devRoot from devicemanager [None] is not correct"
  • bug 697112 - add more twigs
  • bug 689649 - update buildbot config.py to adjust side by side talos staging for mozafterpaint
  • bug 695707 - mozharness should be tagged automatically for 8.0+ releases
in production 20111021 0932 PDT
  • bug 683448 - Permission check and virus scan tests shouldn't fail if files pushed to the releases directory
  • bug 689649 - disable old_suites for mozilla-beta
  • bug 692504 - push betas to internal mirrors automatically
  • bug 693015 - disable android debug tests
  • bug 694077 - add aus2_mobile_* to the "update branch vars loop" in config.py
  • bug 694893 - Bump disk space requirement for codecoverage to 7G
  • bug 695161 - backout 1318d1bbc15a to re-enable Win64 updates
  • bug 695429 - FF8 beta4 config changes
  • bug 696165 - enable tegras 129 - 153
in production 20111019 1100 PDT bug 695525 Pulse enabled on test-master01
in production (build only) 20111017 1728 PDT
  • bug 695161 Disable updates to broken Win x64 builds
in production 20111017 1100 PDT
  • bug 690860 enable android debug nightly on m-c
  • bug 694235 config tests shouldn't fail if there are no try slaves
  • bug 694106 remove tegra try pool
  • bug 676879 Config changes required to run valgrind as a nightly builder
  • bug 694716 patch by joel to fix broken mochitests due to bug 691411
  • bug 694077 Enable nightlies builds and updates for birch branch
in production 20111017 0900 PDT
in production 20111012 0735 PDT
  • backout bug 692928.jhford
  • bug 693903 Update slaves for staging and preproduction configs. rail
  • bug 692823 Reduce PGO sets to 6 hours until bug 691675 is fixed. armenzg
  • bug 693686 PGO talos is submitting to the Firefox-Non-PGO tree. armenzg
in production 20111011 1515 PDT (for build masters, others later
  • bug 693350 - Don't try to add bouncer entries in preproduction
  • bug 692388 - mozharness MercurialVCS with HG_SHARE_BASE_DIR set completely ignores specified revision
  • No Bug, do compare_attrs better for DependentL10n, so we don't throw in dump_masters. Will followup later to get compare_attrs better for all of buildbotcustom. Not used for Firefox builds, so NPOTB
  • bug 693686 - PGO talos builds reporting to Non-PGO branches
  • bug 693794 - remove unneeded usebuildbot=1 from tbpl links in try emails
in production 20111007 1550 PDT
  • bug 692928 turn off rev4 on try
  • bug 692910 Update preproduction test slave list
  • bug 688296 python module conflict with xcode module
  • bug 692646 enable PGO on release builds again
  • bug 692388 mozharness MercurialVCS with HG_SHARE_BASE_DIR set completely ignores specified revision
in production 20111006 1230 PDT
  • bug 681834 Insert finished jobs in the statusdb more frequently
  • bug 686578 SpiderMonkey builds on IonMonkey TBPL - enable all debug spidermonkey builds on ionmonkey
  • bug 687832 create generic RETRY signifier, and make retry.py print it when it fails to successfully run * * bug 692358 Fix log uploading for PGO builds and tests
  • bug 692370 Add branch name to PGO scheduler so that it shows up on self-serve
in production 20111005 1050 PDT
in production 20111005 1821 PDT
  • Backed out: bug 671450 - Use buildid and rev to create tinderbox-builds path (post_upload.py part)
in production 20111005 1632 PDT
  • bug 671450 - Backout log_uploader.py change, as got_revision doesn't exist on test jobs
in production 20111005 1546 PDT
  • bug 671450 - Use buildid and rev to create tinderbox-builds path (buildbot part)
  • bug 686831 - Stop TinderboxPrint-ing the rev early for try
  • bug 691483 - Do 3.6.23 -> 7.0.1 advertised major update
  • bug 689750 - stop sending sendchanges to jhfords personal master
in production 20111005 1526 PDT
  • bug 671450 - Use buildid and rev to create tinderbox-builds path (post_upload.py part)

Archive

Older Changes

Android Testing

Web Server Cluster

Revision Bug # When Who
3efbac1f685a unknown unknown unknown

Update Procedure:

ssh to bm-remote-talos-webhost-01
cd /var/www/html/talos
hg pull && hg up
rsync -azf --delete . bm-remote-talos-webhost-02:/var/www/html/.
rsync -azf --delete . bm-remote-talos-webhost-02:/var/www/html/.

Servers:

  • bm-remote-talos-webhost-01.build.mozilla.org
  • bm-remote-talos-webhost-02.build.mozilla.org
  • bm-remote-talos-webhost-03.build.mozilla.org

clientproxy servers

Production

Talos Rev Pageloader Rev Taras Bench Rev sut_tools Bug # When Who
2a995b4ed124 31249cbe4f19 bfc910cd8dd3 ae5d6911905a talos: bug 629503 20110202 23:00 PDT bear

Pending

Talos Rev Pageloader Rev Taras Bench Rev sut_tools Bug # When Who

Servers:

  • bm-foopy01.build.mozilla.org
  • bm-foopy02.build.mozilla.org
/builds/cp
/builds/talos-data/talos
/builds/talos-data/talos/pageloader@mozilla.org
/builds/talos-data/talos/bench@taras.glek
/builds/sut_tools