ReleaseEngineering/Maintenance

< ReleaseEngineering
Revision as of 11:37, 24 November 2014 by Pmoore (talk | contribs) (reconfig)

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

Non-releng repositories involved in the CI

TBPL

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2014-11-24 03:37 PT

buildbot-configs

mozharness

in production 2014-11-22 16:53 PT

buildbot-configs

mozharness

in production 2014-11-21 14:51 PT

buildbot-configs

  • bug 1092428 - Disable e10s talos suites on holly. Enable e10s talos suites on mozilla-central PGO builds - r=bhearsum (e3fc54de6e7e)
  • bug 1098111 - Remove reftest-ipc and crashtest-ipc - r=catlee (0b1dbaccff51)
  • bug 1099244 - Split reftest into two chunks on linux32 on cedar - r=jlund (4cefbd4f12be)
  • bug 1102283 - figure out firefox 34 release mechanics - add fake mozilla-release-34.1 branch to buildbot-configs for firefox 34 release purposes - r=rail (c31b3686e3f7)
  • bug 1102283 - figure out firefox 34 release mechanics - fix up buildbot-configs - r=rail (b8a3e07b35c6)
  • bug 1102283 - figure out firefox 34 release mechanics - remove verify configs for 34.0.5 to not run update automation for it - r=rail (c7866b0ffaa8)
  • bug 791924 - Enable mozharness pinning on Ash - r=rail (198d3b21d967)
  • No bug - Update release config for Fennec-34.0b11-build1 (4408c9184a1e)
  • No bug - Update release config for Firefox-34.0b11-build1 (47d62deebb61)

buildbotcustom

mozharness

  • bug 1101800 - failed desktop mozharness build balrog updates don't change build status/color - r=mgerva (2fdc1c6ae231)
  • bug 1102489 - m-i and fx-team test logs are being uploaded to ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/11/ - r=catlee (2453fb0cf052)
in production 2014-11-20 12:04 PT

buildbot-configs

mozharness

in production 2014-11-19 02:52 PT

buildbot-configs

  • bug 1054157 - turn off tests on release automation builds - r=bhearsum (22019f9ea388)
  • No bug - Update release config for Fennec-34.0b10-build1 (57128dfcf786)
  • No bug - Update release config for Firefox-34.0b10-build1 (363c9b47d131)
  • No bug - Update release config for Fennec-34.0b10-build2 (789f282aad87)

buildbotcustom

mozharness

in production 2014-11-18 14:00 PT bug 1022763 - Use s3-external-1.amazonaws.com to access S3 from us-east-1
in production 2014-11-17 12:45 PT

buildbot-configs

  • bug 1055918 - switch all fx-team, m-i, m-c, b2g-inbound, and try desktop builders to mozharness mach, r=bhearsum (5a9b98575bc9)

mozharness

  • No bug - Better parsing of hacky diff files as part of gecko migration. r=rail (63a9bfd6568c)
  • bug 1097666 - Tracking bug for 28-Nov-2014 migration work. Don't require require_remove_locales for mozilla-beta->mozilla-release uplift. r=bhearsum (782af6f8590d)
  • bug 1100474 - Do not lock aurora updates by default. r=bhearsum (412fc8f62e17)
in production 2014-11-17 09:01 PT

buildbot-configs

mozharness

in production 2014-11-13 4:00 PT
  • Build and Scheduler masters only
    • bug 1094551 - Support EME voucher signing on signing server
in production 2014-11-10 19:29 PT

mozharness

in production 2014-11-10 15:25 PT bug 1022763 - have mock use repos in s3
in production 2014-11-07 07:29 PT

buildbot-configs

in production 2014-11-07 01:34 PT

buildbot-configs

buildbotcustom

mozharness

  • bug 1055918 - mozharness mach desktop build fixes for nonunified, try upload, and check test - r=mgerva (1c6f07983bb2)
  • bug 1068153 - Determine an output parser to use based on a set of suites using structured logging derived from an in tree config - r=ahal (0a2b30f409c8)
  • bug 1093355 - Artificially limit the browser build to -j4 to control memory usage - r=terrence (72fb76a92c59)
  • bug 1093867 - Chunk mochitest-gl on at least android 2.3 - r=gbrown (2d78545b5039)
  • bug 740142 - Move Firefox Desktop repacks to use mozharness, updated configuration, removed obsolete files - r=jlund (29c32c056cdf)
in production 2014-11-07 6:12 PT

mozharness

in production 2014-11-06 20:12 PT

buildbot-configs

  • No bug - disable tagging for Firefox 34.0b7build1 (6264384558e3)
in production 2014-11-05 06:11 PT

buildbot-configs

buildbotcustom

  • bug 1019724 - Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond) - r=rail (cbc087de6234)

mozharness

in production 2014-11-04 04:37 PT

buildbot-configs

  • bug 1019724 - Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond) - r=nthomas (cee72740386f)
  • bug 1043699 - Add chunks to Mulet reftests - r=jlund (f6d6c9302c55)
  • bug 947462 - Use mozilla-centos6-x86_64 chroots for linux32 b2g builds - r=nthomas (5562ee8c217a)
  • No bug - Update release config for Firefox-31.2.1esr-build1 (1d024e886e40)
  • No bug - Update release config for Fennec-34.0b6-build1 (cb0a62abf673)
  • No bug - Update release config for Firefox-34.0b6-build1 (323b82ea38e2)

mozharness

in production 2014-11-03 19:05 PT

buildbot-configs

in production 2014-10-31 14:49 PT

buildbot-configs

buildbotcustom

mozharness

  • bug 1055918 - mach mozharness - nonunifed fix, uploadsymbols env not passed for try, dont set unset mach prop - r=mgerva (8271db6faea5)
in production 2014-10-30 15:55 PT

buildbotcustom

  • bug 965151 - Fix bustage in valgrind builds from missing toolsdir property, r=Callek
in production 2014-10-30 15:40 PT

buildbot-configs

buildbotcustom

mozharness

in production 2014-10-28 12:51 PT

buildbot-configs

  • bug 1019724 - Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond) - r=rail (da3ac7ace7e3)
  • bug 1019724 - Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond) - r=rail (eab4127639e5)

buildbotcustom

  • bug 1019724 - Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond) - r=rail (0151fbedc670)
  • bug 1019724 - Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond) - r=rail (a2841db14eaf)
  • bug 1087376 - Make gaia-ui-test and gaia-js-integration act as aliases for the related chunks - r=RyanVM (b7c5bdc500cd)

mozharness

  • bug 1019724 - Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond) - r=rail (282d655f1ab2)
  • No bug - Bustage fix for typo - r=bustage (e8616b7fdc43)
in production 2014-10-28 03:27 PT

buildbot-configs

buildbotcustom

  • bug 1087486 - l10n nightly builders shouldn't merge when enable_l10n_onchange isn't set - r=bhearsum (00cd85f4b283)

mozharness

in production 2014-10-27 14:47 PT

buildbot-configs

  • No bug - Fix Firefox 34.0b4 release by specifying correct build# for 34.0b3 (6f62dac88802)
in production 2014-10-24 03:01 PT

buildbot-configs

buildbotcustom

in production 2014-10-22 13:01 PT

buildbot-configs

mozharness

in production 2014-10-21 11:31 PT

buildbot-configs

mozharness

in production 2014-10-21 04:51 PT

buildbot-configs

mozharness

in production 2014-10-21 02:08 PT

buildbot-configs

in production 2014-10-20 13:56 PT

buildbot-configs

in production 2014-10-19 02:00 PT

buildbot-configs

in production 2014-10-17 13:12 PT

buildbot-configs

in production 2014-10-17 07:34 PT

buildbot-configs

mozharness

in production 2014-10-16 19:51 PT

buildbot-configs

in production 2014-10-16 12:51 PT

buildbot-configs

in production 2014-10-16 09:08 PT

buildbot-configs

mozharness

in production 2014-10-15 12:53 PT

buildbot-configs

buildbotcustom

mozharness

  • bug 1055918 - mozharness mach builds, accept warning return code from mach build, fatal on error - r=mgerva (8a6e8b1bfe86)
in production 2014-10-15 07:19 PT

buildbot-configs

in production 2014-10-14 05:46 PT
  • Backed out changeset 82fa2f13a546
  • Backed out changeset fb020d38b515
in production 2014-10-13 09:11 PT
  • Update release config for Fennec-31.2.0esr-build2
  • Update release config for Fennec-33.0-build2
  • Update release config for Firefox-31.2.0esr-build2
  • Update release config for Firefox-31.2.0esr-build3
  • Update release config for Firefox-33.0-build2
  • Update release config for Thunderbird-31.2.0-build2
  • bug 1061861 - Tracking bug for 13-oct-2014 migration work.
  • bug 1077154 - android-x86 and android are both uploading the to the same symbols path on mozilla-aurora
  • bug 1080134 - Disable VM tests; enable talos tests.
  • bug 885331 - added PYTHONPATH in doPartnerRepacks env.
in production 2014-10-13 07:15 PT
  • bug 1061589 - Rename of ffxbld_dsa to ffxbld_rsa in puppet/hiera
in production 2014-10-10 15:40 PT

buildbot-configs

in production 2014-10-09 12:34 PT
  • bug 1078507 - undo unintended change to Makefile.master -
in production 2014-10-08 19:45 PT
  • Complete backout of bug 1079004 (wasn't merged from default for some reason)
in production 2014-10-08 07:19 PT
  • Backed out changeset 180e48cf37f5 now * bug 1078820 - is fixed
  • Update release config for Fennec-33.0b10-build1
  • Update release config for Firefox-31.2.0esr-build1
  • Update release config for Firefox-33.0-build1
  • Update release config for Thunderbird-31.2.0-build1
  • bug 1035226 - Only use RUNTIME iff Mule
  • bug 1035226 - Pass RUNTIME to Gij for Mule
  • bug 1035226 - Schedule Gij on Mulet on cedar
  • bug 1043699 - Enable Mulet reftests on Cedar.
  • bug 1072111 - , disable flame (jb) for B2G 2.0 and higher
  • bug 1074508 - Point osx mulet builds to new mozconfig
  • bug 1075607 - create mozilla-b2g34_v2_1 branch.
  • bug 1078248 - android single locale nightly builds need to retry balrog submissions
  • bug 1079004 - , backout f2f8e93ea310 now bug 1078820 is fixed
  • bug 1079004 - Work around unreachable symbolpush.m.o
  • bug 650989 - Add tests and build urls to emulator buildbot properties
in production 2014-10-06 21:30 PT
in production 2014-10-06 07:40 PT
  • bug 1075675 - copy google oauth api key into mock environment.
  • bug 1075712 - support signing 64-bit windows signing in buildbot-configs.
in production 2014-10-06 05:14 PT
  • bug 1061861 - Tracking bug for 13-oct-2014 migration work. Bump mozilla-release version.
in production 2014-10-06 04:05 PT
  • bug 1056837 - , generate 33.0b9 -> 33.0 partial updates
  • bug 1074147 - Part 2: On holly add mochitest plain tests with the content sandbox for win32.
  • bug 1075607 - create mozilla-b2g34_v2_1 branch.
  • bug 1077374 - Firefox 33 beta builds are not picking up changes.
  • bug 1077456 - disable l10n repacks on oak
in production 2014-10-03 09:37 PT
  • Update release config for Firefox-33.0b9-build1
  • bug 1043699 - Enable mulet reftests on try.
  • bug 1052916 - Retry bouncer submissions.
  • bug 1066823 - remove old code that supports tegras in buildbot-configs/buildbotcustom/tools/mozharness/puppe
  • bug 1067628 - Schedule win32 Mulet builds on cedar
  • bug 1071310 - Disable e10s Talos tests on winxp
  • bug 1072165 - Increase space requirement for TB Linux* release build.
  • bug 1074147 - Part 1: Add option for --content-sandbox to desktop_unittest.py to be passed onto mochitest suites,r=armenzg
  • bug 1075787 - Make Gij take passed/failed test counts into account when setting tbpl status
  • bug 1075992 - Read in-tree configs if it has not been set through download-and-extract.
  • bug 1077432 - Skip more unittests on capacity-starved platforms -