Build:OutageReports:20070410-02: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| Line 1: | Line 1: | ||
== Outage Template == | == Outage Template == | ||
On Apr 10th from ~ 4 am, moz180-linux-tbox | On Apr 10th from ~ 4 am, moz180-linux-tbox experienced a service outage for ~ 10 hours. | ||
'''What was affected:''' | '''What was affected:''' | ||
| Line 14: | Line 14: | ||
Some ideas | Some ideas | ||
* Monitor tinderbox disk space (eg Nagios) | |||
* Make tinderbox go red on symbol problems (creation and push) | |||
Latest revision as of 21:26, 10 April 2007
Outage Template
On Apr 10th from ~ 4 am, moz180-linux-tbox experienced a service outage for ~ 10 hours.
What was affected:
Firefox linux builds on Mozilla1.8.0 branch (en-US and locales)
What was the cause of the outage:
The /builds partition was full. It looks like symbol generation has been busted for the last few days (at least), with firefox-bin.elf ending up a random, truncated, size and zero byte files for the other executables.
What will be done to prevent this in the future:
Some ideas
- Monitor tinderbox disk space (eg Nagios)
- Make tinderbox go red on symbol problems (creation and push)