ReleaseEngineering/Maintenance

< ReleaseEngineering
Revision as of 11:12, 31 July 2017 by Mtabara (talk | contribs) (reconfig)

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our master buildbots.

Relevant repositories

Non-releng repositories involved in the CI

Treeherder

  • Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
  • [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2017-07-31 04:12 PT

buildbot-configs

  • bug 1369535 - m-r gecko version bump bbconfigs - r=jlorenzo (acbb8cf0729e)
  • No bug - No Bug - Staging release: Point to a disposable balrog instance - r=rail (c700658593c4)
  • No bug - No Bug - Staging release: Point to rail's disposable repo - r=jlorenzo (749272eda177)
in production 2017-06-12 03:31 PT

buildbot-configs

in production 2017-06-05 06:32 PT

buildbot-configs

in production 2017-06-05 02:12 PT

buildbot-configs

in production 2017-05-11 06:12 PT

buildbot-configs

in production 2017-03-16 07:25 PT

buildbot-configs

in production 2017-03-10 15:02 PT

buildbot-configs

in production 2017-03-10 10:11 PT

buildbot-configs

in production 2017-03-09 13:18 PT

buildbot-configs

in production 2017-03-06 11:57 PT

buildbot-configs

in production 2017-03-06 11:49 PT

buildbot-configs

in production 2017-03-03 13:50 PT

buildbot-configs

in production 2017-03-03 12:53 PT

buildbot-configs

in production 2017-03-01 11:50 PT

buildbot-configs

in production 2017-02-27 19:34 PT

buildbotcustom

in production 2017-02-27 18:50 PT

buildbot-configs

  • No bug - Update release config for Fennec-52.0b10-build1 (edf97f3ab377)

buildbotcustom

in production 2017-02-27 13:40 PT

buildbot-configs

in production 2017-02-27 11:56 PT

buildbot-configs

in production 2017-02-24 13:02 PT

buildbotcustom

in production 2017-02-23 06:27 PT

buildbot-configs

in production 2017-02-17 09:50 PT

buildbot-configs

in production 2017-02-16 13:27 PT

buildbot-configs

  • bug 1338610 - Split up mochitest-other into separate mochitest-a11y and mochitest-chrome jobs - r=jlund (4f814a83ba4f)
  • bug 1339151 - buildbot configs to use different tasks for signed and unsigned bits in release promotion - r=jlund (fb545d14c97f)
in production 2017-02-15 12:50 PT

buildbot-configs

  • No bug - backouts Bug No Bug - rev: c9b1f1f58974 (fbcd05687641)
  • No bug - reverts 51.0.4 release fennec patch (7d7513aadda6)
in production 2017-02-08 13:10 PT

buildbot-configs

in production 2017-02-08 12:19 PT

buildbot-configs

buildbotcustom

in production 2017-01-23 03:51 PT

buildbot-configs

in production 2017-01-05 04:33 PT

buildbot-configs

in production 2016-12-29 06:18 PT

buildbot-configs

  • bug 1306252 - Enable nightly/tinderbox builds for kab on comm-aurora/beta - r=rail (b6acca53ba05)
  • No bug - Update release config for Fennec-51.0b9-build1 (86dd645e79a5)
  • No bug - Update release config for Thunderbird-45.6.0-build2 (0b2f910429ca)
in production 2016-12-19 08:26 PT

buildbot-configs

  • bug 1176784 - reconfigs should not rely on seta server - r=kmoir (c61b82cda2ed)
  • bug 1324290 - Sign the comm-central win64 builds - r=rail (3ba60d3917ff)
  • No bug - Update release config for Fennec-51.0b4-build1 (ab541569e70a)
  • No bug - Update release config for Fennec-50.0.2-build1 (4cf29de987a0)
  • No bug - Update release config for Fennec-51.0b5-build1 (1e0a5b9d2fd0)
  • No bug - Update release config for Thunderbird-45.5.1-build1 (952bae652661)
  • No bug - Update release config for Thunderbird-45.5.1-build2 (5c8169fadf9d)
  • No bug - updating seta data for 2016-12-01 (71bbb16bb232)
  • No bug - Update release config for Thunderbird-51.0b1-build2 (9fe6bfabf073)
  • No bug - Update release config for Fennec-50.1.0-build1 (f6267b86bcfd)
  • No bug - Update release config for Fennec-50.1.0-build2 (a26d58ceca67)
  • No bug - Update release config for Thunderbird-51.0b1-build3 (228acd65b6b9)
  • No bug - Update release config for Thunderbird-51.0b1-build4 (07a7e88052fb)
  • No bug - Update release config for Fennec-51.0b7-build1 (f118db65e1d5)
  • No bug - Update release config for Thunderbird-45.6.0-build1 (e694dd5b252d)
in production 2016-11-21 09:32 PT

buildbot-configs

in production 2016-11-16 13:34 PT

buildbot-configs

  • No bug - Update release config for Thunderbird-51.0b1-build1 (c28939dcc685)
  • No bug - Update release config for Fennec-51.0b1-build2 (ff67fea64ddf)
  • No bug - Update release config for Fennec-51.0b1-build3 (4a8f906be2ba)

buildbotcustom

in production 2016-11-14 04:42 PT

buildbot-configs

in production 2016-11-07 13:00 PT

buildbotcustom

in production 2016-10-31 12:07 PT

buildbot-configs

in production 2016-10-27 11:38 PT

buildbot-configs

in production 2016-10-25 19:00 PT

buildbot-configs

in production 2016-10-25 09:24 PT

buildbot-configs

in production 2016-10-24 17:58 PT

buildbot-configs

in production 2016-10-21 06:56 PT

buildbot-configs

  • bug 1309894 - Use "poll_all_branches" in configs to explicitly set polling behaviour - r=catlee (f3d8389cb9bf)
  • No bug - Update release config for Fennec-49.0.2-build2 (08de42085241)

buildbotcustom

in production 2016-10-18 08:54 PT

buildbotcustom

in production 2016-10-13 10:14 PT

buildbot-configs

  • bug 1308789 - Set the Win64 builds to be the default try_by_default: True, because we want them to build by default, it's only the Win8 tests that are supposed to be try_by_default: False - r=catlee (a9bbcd872ad9)
  • bug 1309894 - Investigate shipping old release version off of jamun - r=Callek (6f25921852c3)
  • No bug - Update release config for Fennec-50.0b6-build1 (f1f2fc3fa06e)
in production 2016-10-10 07:54 PT

buildbot-configs

  • bug 1292656 - decommission another 10 t-snow-r4 machines - r=coop (02b3c211cc48)
  • bug 1300413 - Use unbuffered python in case we're missing output - r=Callek (1d22c76665b8)
  • No bug - Update release config for Thunderbird-45.4.0-build1 (a4d817817321)
  • No bug - Update release config for Thunderbird-50.0b1-build1 (ab02ce360da9)
  • No bug - Update release config for Fennec-50.0b4-build1 (f854aac09b67)
  • No bug - Update release config for Thunderbird-50.0b1-build2 (acc59373912b)
in production 2016-09-21 10:45 PT

buildbot-configs

  • No bug - Update release config for Fennec-50.0b1-build1 (d17d80076abd)
in production 2016-09-19 22:09 PT

buildbot-configs

in production 2016-09-19 06:37 PT

buildbot-configs

in production 2016-09-14 13:31 PT

buildbot-configs

in production 2016-09-06 10:59 PT

buildbot-configs

in production 2016-09-05 07:59 PT

buildbot-configs

in production 2016-08-26 11:33 PT

buildbot-configs

buildbotcustom

in production 2016-08-25 10:56 PT

buildbot-configs

  • bug 1298013 - Migrate remaining win 7 mochitest bc jobs to AWS - r=catlee (7c1e748101f6)
  • No bug - Update release config for Thunderbird-49.0b1-build4 (26a6a5f90516)
  • No bug - Update release config for Thunderbird-45.3.0-build1 (4b44a217b2ef)

buildbotcustom

in production 2016-08-23 12:23 PT

buildbot-configs

in production 2016-08-23 10:14 PT

buildbot-configs

in production 2016-08-19 05:55 PT

buildbot-configs

buildbotcustom

in production 2016-08-11 13:16 PT

buildbotcustom

in production 2016-08-10 13:50 PT

buildbot-configs

buildbotcustom

in production 2016-08-08 13:15 PT

buildbot-configs

in production 2016-08-04 09:11 PT

buildbot-configs

  • bug 1288043 - migrate win7 gpu jobs to gfx instances - r=rail (199eca90a0a9)
  • bug 1291044 - Use 10 chunks for wpt on win 7 vm debug to prevent timeouts - r=jmaher (c23e2a8603f5)
  • No bug - [PATCH] Bug 1271153 - Adjust automation for the version_display.txt new file on tb - r=rail (06d19bac0eff)
in production 2016-08-01 06:14 PT

buildbot-configs

in production 2016-07-28 04:56 PT

buildbotcustom

in production 2016-07-27 09:54 PT

buildbot-configs

  • No bug - Update release config for Fennec-48.0b11-build1 (9c1804c74761)
  • No bug - Update release config for Fennec-48.0-build1 (49e72329db05)
in production 2016-07-25 07:30 PT

buildbot-configs

in production 2016-07-22 09:39 PT

buildbot-configs

  • bug 1233953 - Intermittent Linux debug command timed out: 7200 seconds elapsed running - r=rail (2f53b99fd268)
  • bug 1282849 - disable fennec debug builds and tests in buildbot - r=kmoir (19f3d53951ae)
  • No bug - Update release config for Fennec-48.0b8-build1 (da62abc85e9d)
  • No bug - Update release config for Fennec-48.0b9-build1 (2b19aa948928)
  • No bug - Update release config for Fennec-48.0b10-build1 (ad5f8c375bcd)
in production 2016-07-12 16:12 PT

buildbot-configs

  • No bug - Update release config for Thunderbird-48.0b1-build2 (205cec6585a1)

buildbotcustom

in production 2016-07-07 11:34 PT

buildbot-configs

buildbotcustom

in production 2016-06-27 18:57 PT

buildbot-configs

in production 2016-06-07 13:55 PT

buildbotcustom

  • bug 1262760 - Use the objdir's virtualenv's python executable. -- fixes typo - r=rail (hopefully) post-landing (6f6a9824ca4d)
in production 2016-06-07 11:56 PT

buildbotcustom

in production 2016-06-02 08:45 PT

buildbot-configs

in production 2016-05-30 06:58 PT

buildbot-configs

in production 2016-05-29 14:55 PT

buildbot-configs

in production 2016-05-27 12:05 PT

buildbot-configs

in production 2016-05-27 09:48 PT

buildbot-configs

in production 2016-05-27 07:55 PT

buildbot-configs

in production 2016-05-27 05:59 PT

buildbotcustom

  • No bug - Backed out changeset 899b41a5e8e5, for xpcshellcommandline bustage (c543cd26f4ad)
in production 2016-05-27 05:51 PT

buildbot-configs

in production 2016-05-26 01:20 PT

buildbot-configs

buildbotcustom

in production 2016-05-24 10:56 PT

buildbot-configs

in production 2016-05-23 09:45 PT

buildbot-configs

in production 2016-05-20 08:37 PT

buildbot-configs

in production 2016-05-19 10:47 PT

buildbot-configs

in production 2016-05-17 09:46 PT

buildbot-configs

in production 2016-05-16 11:22 PT

buildbot-configs

buildbotcustom

in production 2016-05-16 08:15 PT

buildbot-configs

buildbotcustom

in production 2016-05-12 08:43 PT

buildbot-configs

buildbotcustom

in production 2016-05-12 02:03 PT

buildbot-configs

  • bug 1253341 - Run duplicate Talos jobs in AWS for Linux (reverted) - r=Callek (2302dc47cb21)
  • bug 1268542 - Disable Linux64 Debug builds and tests in buildbot - r=kmoir (cfe4c07f8161)
  • bug 1271550 - Add new 10.10.5 machines to releng configs (750ab5df5a06)
  • No bug - Update release config for Thunderbird-45.1.0-build2 (48e0db6d4d22)
  • No bug - Update release config for Thunderbird-45.1.0-build3 (c199b13b84a3)
  • No bug - Backed out changeset c199b13b84a3 to let TB 45.1.0 build3 to restart (a55a7e24bba1)
  • No bug - Update release config for Thunderbird-45.1.0-build3 (3a7c1d3dbace)
  • No bug - Update release config for Fennec-47.0b4-build1 (f7b8a43eed4f)

buildbotcustom

in production 2016-05-06 09:58 PT

buildbot-configs

buildbotcustom

in production 2016-05-02 13:55 PT

buildbotcustom

in production 2016-05-02 13:00 PT

buildbot-configs

buildbotcustom

in production 2016-04-27 10:34 PT

buildbot-configs

in production 2016-04-19 06:35 PT

buildbot-configs

buildbotcustom

in production 2016-04-19 05:32 PT

buildbotcustom

in production 2016-04-13 06:07 PT

buildbot-configs

buildbotcustom

in production 2016-04-04 13:40 PT

buildbot-configs

buildbotcustom

in production 2016-03-29 13:32 PT

buildbot-configs

buildbotcustom

in production 2016-03-21 17:41 PT

buildbotcustom

in production 2016-03-21 14:15 PT

buildbot-configs

buildbotcustom

in production 2016-03-18 12:37 PT

buildbotcustom

in production 2016-03-16 10:35 PT

buildbot-configs

  • bug 1252292 - Don't run Ru-e10s on debug builds - r=me (04c8fe62d3a4)
  • bug 1253341 - Enable talos on linux64 AWS machines on try - r=kmoir (2d6158d681e0)
  • bug 1257230 - Incorrect branding for 45.0.1esr build - r=jlund (b5163b4a7814)
  • No bug - Update release config for Fennec-46.0b2-build1 (216ac6505867)
  • No bug - Update release config for Firefox-38.7.1esr-build1 (2b0b8272dda2)
  • No bug - backouts release change for 38.7.1esr - rev: 2b0b8272dda2 (ea85259cf436)
  • No bug - merges backout 2b0b8272dda2 -> default (7b0276c224b4)
  • No bug - Update release config for Firefox-38.7.1esr-build1 (b735b52c0d5a)
  • No bug - Update release config for Firefox-45.0.1-build1 (b3267390eb7d)
  • No bug - Update release config for Firefox-45.0.1esr-build1 (1f7801999788)
  • No bug - Update release config for Fennec-45.0.1-build1 (e759c66d311c)
  • No bug - Backed out 2d6158d681e0 for suspicion of causing intermittent checkconfig (7c463958098d)

buildbotcustom

  • bug 1253341 - Allow running talos on AWS machines - r=kmoir (42184dbdf8f3)
  • No bug - Backed out changeset 42184dbdf8f3 for suspicion of causing intermittent checkconfig failureis (046c46eb099d)
in production 2016-03-15 07:43 PT

buildbot-configs

in production 2016-03-11 09:50 PT

buildbot-configs

buildbotcustom

in production 2016-03-10 06:17 PT

buildbot-configs

in production 2016-03-09 06:14 PT

buildbot-configs

in production 2016-03-07 19:31 PT

buildbot-configs

in production 2016-03-07 16:12 PT

buildbot-configs

in production 2016-03-07 05:36 PT

buildbot-configs

in production 2016-03-04 13:23 PT

buildbot-configs

in production 2016-03-04 12:26 PT

buildbot-configs

in production 2016-03-04 09:03 PT

buildbot-configs

in production 2016-03-04 07:06 PT

buildbot-configs

in production 2016-03-03 16:54 PT

buildbot-configs

in production 2016-03-02 13:51 PT

buildbot-configs

buildbotcustom

  • No bug - No Bug - shorten travis results notification to be less spammy - r=catlee (4b96f5cb0d0f)
in production 2016-03-02 06:25 PT

buildbot-configs

in production 2016-03-01 07:30 PT

buildbot-configs

in production 2016-02-29 08:10 PT

buildbotcustom

in production 2016-02-25 09:50 PT

buildbot-configs

buildbotcustom

in production 2016-02-24 13:45 PT

buildbot-configs

in production 2016-02-23 13:53 PT

buildbot-configs

in production 2016-02-22 13:17 PT

buildbot-configs

  • No bug - Update release config for Firefox-45.0b8-build1 (5f9a37958eaf)
  • No bug - Update release config for Firefox-45.0b8-build1 (5f9a37958eaf)

buildbotcustom

in production 2016-02-19 13:00 PT

buildbotcustom

in production 2016-02-19 07:03 PT

buildbot-configs

buildbotcustom

in production 2016-02-17 08:46 PT

buildbot-configs

  • bug 1246627 - Schedule e10s jsreftests on win7 patch by aselagea - r=jgriffin (cc37bdb15492)
  • No bug - revert Bug 1246627 - Schedule e10s jsreftests on win7 (71b9ccb8abf1)
  • No bug - Update release config for Fennec-45.0b6-build1 (69f84fb1f35b)
  • No bug - Update release config for Firefox-45.0b6-build1 (df4ed20df968)

buildbotcustom

in production 2016-02-14 08:43 PT

buildbot-configs

buildbotcustom

  • bug 1244510 - Add a try_parser alias for mochitest-e10s-(bc|browser-chrome) patch by aselagea - r=kmoir (c994ec64ba03)
in production 2016-02-12 07:43 PT

buildbot-configs

  • bug 1218589 - Nightly mozilla-central builds of graphene patch by aselagea - r=kmoir (f88d45c3fbe6)
  • bug 1229532 - remove hg bundle builder config - r=bhearsum (f0c9c5b5a7b8)
  • bug 1246627 - Schedule e10s jsreftests on win7 patch by aselagea - r=kmoir (66bc120b896c)
  • bug 787208 - In support of merging comm-central and mozilla-central together, configure cypress as a Thunderbird branch. r=rail - r=Fallen f=jcranmer (b92aaf935173)
  • No bug - Update release config for Firefox-38.6.1esr-build1 (33fe33d65478)
  • No bug - Update release config for Firefox-44.0.2-build3 (09cd8415adb9)
  • No bug - Update release config for Thunderbird-38.6.0-build1 (91cf6aa50268)
  • No bug - Update release config for Firefox-45.0b5-build1 (6376fd9567c4)

buildbotcustom

in production 2016-02-02 14:53 PT

buildbot-configs

  • bug 1242103 - Add browser-chrome-screenshots to m-c and other try - r=armenzg (63614b6564ee)
  • No bug - Update release config for Fennec-45.0b2-build1 (7a121e3abe60)
  • No bug - Update release config for Firefox-45.0b2-build1 (a70a9918a489)
  • No bug - Update release config for Thunderbird-45.0b1-build1 (39642e4c3ba4)
in production 2016-01-26 20:05 PT

buildbotcustom

buildbot-configs

in production 2016-01-26 18:05 PT

buildbotcustom

in production 2016-01-26 16:00 PT

buildbotcustom

buildbot-configs

in production 2016-01-25 20:47 PT

buildbotcustom

in production 2016-01-25 14:54 PT

buildbotcustom

in production 2016-01-25 05:50 PT

buildbot-configs

in production 2016-01-21 12:12 PT

buildbot-configs

in production 2016-01-21 10:56 PT

buildbot-configs

buildbotcustom

in production 2016-01-20 06:16 PT

buildbot-configs

  • No bug - Update release config for Fennec-44.0b11-build1 (370703910927)

buildbotcustom

in production 2016-01-19 07:36 PT

buildbot-configs

buildbotcustom

in production 2016-01-18 06:19 PT

buildbot-configs

mozharness

in production 2016-01-15 08:55 PT

buildbot-configs

  • No bug - Update release config for Fennec-44.0b9-build1 (15dc357fc5bd)
  • No bug - Update release config for Firefox-44.0b9-build1 (df3ee52930d6)

buildbotcustom

in production 2016-01-13 15:03 PT

buildbot-configs

buildbotcustom

in production 2016-01-12 09:04 PT

buildbot-configs

buildbotcustom

mozharness

in production 2016-01-11 08:10 PT

buildbot-configs

buildbotcustom

in production 2016-01-11 07:41 PT

buildbot-configs

buildbotcustom

mozharness

in production 2016-01-07 06:39 PT

buildbot-configs

buildbotcustom