Firefox/Input/Releases/1.8/PostMortem

What Action was Performed?

• Update script
  • push
  • update vendor libs
  • apache reload
  • run crons

Issues before Release

• Stage was displaying a blank white screen 1 out of every 40 or 50 page loads.
• I didn't realize tests haven't been run since the 19th, on staging :-(
  • http://sm-hudson01.mozilla.org:8080/job/input.stage.mozilla.com/

• Moved to Phoenix 2 weeks ago
• Jeremy Orem did a push on 6/21 thinking everything was fine. We had cancelled the push 1/2 an hour before. We had to quickly revert.
• Unsure about latest code commit. Belief was that everything was checked in and tagged.
• Fred was at Djangocon for a week
• Ryan was predisposed with emergency family stuff for a week and Socorro Work Week
• Dave was incredibly hammered with AMO
• Stephen was busy with Socorro, mozilla.com, mobile stuff for the past 2 weeks

Issues during Release

Production

• bug 600445 - Auto-Select default Version when changing Product
• bug 600448 - Mobile "--all--" option shouldn't include Firefox-desktop-build submissions
• bug 600627 - Input Production updated, but not to most recent Commit

Staging

• bug 600444 - Input staging/mobile staging are down for the count
• bug 595358 - Need DB dump from Input prod -> staging (I forgot that was open)

Perspectives

stephend

• Staging should have been up and should have been checked
• Created a release checklist
• No monitors: Nagios and Hudson and both would have caught this

ryansnyder

• lack of preparation
• end of quarter goals push tension, lack-of-focus and tiredness

davedash

• late pushes at the end of the day when people are unavailable
• push was pretty successful just with the issues when people were fresh

aakashd

• lack of a push checklist
• move up the start time

Solutions

• Move releases to 4pm to have availability of a dev who's 'fresh'
• Create a tailored release checklist