Releases/Firefox 3.0.6/Post Mortem: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(New page: The following are the meeting notes from the Firefox 3.0.6 post-mortem held on Friday, February 6, 2009 at 12:00pm PST. == Development == # [issue here] == QA == # [issue here] == Build...)
 
 
(3 intermediate revisions by the same user not shown)
Line 11: Line 11:


== IT ==
== IT ==
# One of bouncer's db servers wasn't running, which caused updates to fail ({{bug|476753}}. Had to pull/push updates a few times while trying to get it working. Why was the server not up and running? How do we ensure this doesn't happen in the future?
# Bouncer slave database was disabled, causing updates to fail when the master was under high load ({{bug|476753}}). Had to pull/push updates a few times while debugging.
# Had to throttle bits after release ({{bug|476875}} because mirrors couldn't handle load. Why? How do we ensure this doesn't happen in the future?
## <code>tm-bouncer01-slave02</code> was disabled in the load balancer during a previous maintenance window and never re-enabled.
## Actions:
### Need to monitor backend service status on the load balancer ({{bug|476764}}) in the same way we monitor origin web servers.  Nagios would have alerted after maintenance window that the slave was disabled.
### Bouncer needs three databases to withstand a failure of one during release ({{bug|477183}}).
# Had to throttle bits after release ({{bug|476875}}) because mirrors couldn't handle load.  
## Mirror weighting needs to automatically change when mirrors come and go.  See {{bug|454023}} ("Sentry should automatically decrease weights on mirrors that get dropped and re-added") for more details.
## Process for adding new mirrors gated on one person, IT's added more subscribers to the list
# Throttling remained on long after necessary
## Need some monitoring to alert/remind when throttling is still configured ({{bug|477184}}).


== Websites ==
== Websites ==
# [issue here]
# [issue here]

Latest revision as of 04:27, 6 February 2009

The following are the meeting notes from the Firefox 3.0.6 post-mortem held on Friday, February 6, 2009 at 12:00pm PST.

Development

  1. [issue here]

QA

  1. [issue here]

Build

  1. [issue here]

IT

  1. Bouncer slave database was disabled, causing updates to fail when the master was under high load (bug 476753). Had to pull/push updates a few times while debugging.
    1. tm-bouncer01-slave02 was disabled in the load balancer during a previous maintenance window and never re-enabled.
    2. Actions:
      1. Need to monitor backend service status on the load balancer (bug 476764) in the same way we monitor origin web servers. Nagios would have alerted after maintenance window that the slave was disabled.
      2. Bouncer needs three databases to withstand a failure of one during release (bug 477183).
  2. Had to throttle bits after release (bug 476875) because mirrors couldn't handle load.
    1. Mirror weighting needs to automatically change when mirrors come and go. See bug 454023 ("Sentry should automatically decrease weights on mirrors that get dropped and re-added") for more details.
    2. Process for adding new mirrors gated on one person, IT's added more subscribers to the list
  3. Throttling remained on long after necessary
    1. Need some monitoring to alert/remind when throttling is still configured (bug 477184).

Websites

  1. [issue here]
Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Firefox_3.0.6/Post_Mortem&oldid=127620"