Firefox/Input/Releases/1.8/PostMortem: Difference between revisions
Jump to navigation
Jump to search
| (16 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
== What Action was Performed? == | |||
* Update script | |||
** push | |||
** update vendor libs | |||
** apache reload | |||
** run crons | |||
== Issues before Release == | == Issues before Release == | ||
* Stage was displaying a blank white screen 1 out of every 40 or 50 page loads. | * Stage was displaying a blank white screen 1 out of every 40 or 50 page loads. | ||
* I didn't realize tests haven't been run since the 19th, on staging :-( | |||
** http://sm-hudson01.mozilla.org:8080/job/input.stage.mozilla.com/ | |||
* Moved to Phoenix 2 weeks ago | |||
* Jeremy Orem did a push on 6/21 thinking everything was fine. We had cancelled the push 1/2 an hour before. We had to quickly revert. | |||
* Unsure about latest code commit. Belief was that everything was checked in and tagged. | |||
* Fred was at Djangocon for a week | |||
* Ryan was predisposed with emergency family stuff for a week and Socorro Work Week | |||
* Dave was incredibly hammered with AMO | |||
* Stephen was busy with Socorro, mozilla.com, mobile stuff for the past 2 weeks | |||
== Issues during Release == | == Issues during Release == | ||
Production | |||
* {{bug|600445}} - Auto-Select default Version when changing Product | * {{bug|600445}} - Auto-Select default Version when changing Product | ||
* {{bug|600448}} - Mobile "--all--" option shouldn't include Firefox-desktop-build submissions | * {{bug|600448}} - Mobile "--all--" option shouldn't include Firefox-desktop-build submissions | ||
* {{bug|600627}} - Input Production updated, but not to most recent Commit | |||
Staging | |||
* {{bug|600444}} - Input staging/mobile staging are down for the count | |||
* {{bug|595358}} - Need DB dump from Input prod -> staging (I forgot that was open) | |||
==== | == Perspectives == | ||
=== | === stephend === | ||
* Staging should have been up and should have been checked | |||
* Created a release checklist | |||
* No monitors: Nagios and Hudson and both would have caught this | |||
=== | === ryansnyder === | ||
* lack of preparation | |||
* end of quarter goals push tension, lack-of-focus and tiredness | |||
=== | === davedash === | ||
* late pushes at the end of the day when people are unavailable | |||
* push was pretty successful just with the issues when people were fresh | |||
=== | === aakashd === | ||
* lack of a push checklist | |||
* move up the start time | |||
==== | == Solutions == | ||
* Move releases to 4pm to have availability of a dev who's 'fresh' | |||
* Create a tailored release checklist | |||
Latest revision as of 20:30, 30 September 2010
What Action was Performed?
- Update script
- push
- update vendor libs
- apache reload
- run crons
Issues before Release
- Stage was displaying a blank white screen 1 out of every 40 or 50 page loads.
- I didn't realize tests haven't been run since the 19th, on staging :-(
- Moved to Phoenix 2 weeks ago
- Jeremy Orem did a push on 6/21 thinking everything was fine. We had cancelled the push 1/2 an hour before. We had to quickly revert.
- Unsure about latest code commit. Belief was that everything was checked in and tagged.
- Fred was at Djangocon for a week
- Ryan was predisposed with emergency family stuff for a week and Socorro Work Week
- Dave was incredibly hammered with AMO
- Stephen was busy with Socorro, mozilla.com, mobile stuff for the past 2 weeks
Issues during Release
Production
- bug 600445 - Auto-Select default Version when changing Product
- bug 600448 - Mobile "--all--" option shouldn't include Firefox-desktop-build submissions
- bug 600627 - Input Production updated, but not to most recent Commit
Staging
- bug 600444 - Input staging/mobile staging are down for the count
- bug 595358 - Need DB dump from Input prod -> staging (I forgot that was open)
Perspectives
stephend
- Staging should have been up and should have been checked
- Created a release checklist
- No monitors: Nagios and Hudson and both would have caught this
ryansnyder
- lack of preparation
- end of quarter goals push tension, lack-of-focus and tiredness
davedash
- late pushes at the end of the day when people are unavailable
- push was pretty successful just with the issues when people were fresh
aakashd
- lack of a push checklist
- move up the start time
Solutions
- Move releases to 4pm to have availability of a dev who's 'fresh'
- Create a tailored release checklist