Firefox/Planning/2011-10-19: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 45: Line 45:


===Firefox Sync===
===Firefox Sync===
* Firefox 7:
;Sync issues - The 503 storm of 2011
** Sync issues - The 503 storm of 2011
* '''Fuel for the fire:'''  
*** '''Fuel for the fire:''' A higher load caused by Instant Sync, no extra capacity for existing users, backoff code was broken, purging of stale data hadn't been done in a while. A db migration was not fully complete. Sync services running on old HW. Only client side testing. There was latency in moving into action made problem more challenging to tackle
** Instant Sync:
*** '''Firefighting:''' Activating backoff code, turns out it was broken.  Many user migrations, brought more DBs online, and some tuning of webservers,load balancers, application stack, kernel tcp params, etc. over several days brought down the fire.
***Created unexpected 30-50% spike in traffic
*** '''Fireproofing:''' Ops is updating metrics to measure capacity so it gives a better picture. Eng-Ops process updated so that an audit of changes happens a week before the channel merges. Backoff code being fixed along with various enhancements for Instant Sync.  Better client and server testing when a change to server load profile is being implemented. Adding an Ops Review and sign off at the feature page level before implementation begins.
** Broken Server-Sent backoff  
** Poor server capacity/health for existing users


* '''Firefighting:'''
** Infrastructure changes
*** DB updates
*** Load balancer updates
*** Webserver tuning
** Migration
*** Added capacity
*** Pruned stale data
*** Migrate users from overloaded servers
* '''Fireproofing:'''
** Better modeling for current and expected traffic on Sync features
** Ops review and sign off on feature pages
** Required system-review for all client behavior changes
** Require reliability metrics and dashboard of these on all channels
** Instill a review of changes and metrics 1 week before merging on all channels
** Fix backoff code in Fx8
** Re-triage of maintenance bugs for any lurking issues
** Blog posts coming from both Ops and Engineering
   
   
* Firefox 10+:
* Firefox 10+:

Revision as of 16:54, 19 October 2011

« previous week | index | next week »

Planning Meeting Details

  • Wednesdays - 11:00am PDT, 18:00 UTC
  • Mountain View Offices: Warp Core Conference Room
  • Toronto Offices: Fin du Monde Conference Room
  • irc.mozilla.org #planning for backchannel
  • (the developer meeting takes place on Tuesdays)

Video/Teleconference Details - NEW

  • 650-903-0800 or 650-215-1282 x92 Conf# 95312 (US/INTL)
  • 1-800-707-2533 (pin 369) Conf# 95312 (US)
  • Vidyo Room: Warp Core
  • Vidyo Guest URL
REMEMBER
These notes are read by people who weren't able to attend the meeting. Please make sure to include links and context so they can be understood.


Actions from Last Week

  • Jen to come back with postmortem summary for sync issues in FF7

Schedule & Progress on Upcoming Releases

Firefox Desktop

Release (3.6, 7)

Beta (8)

Aurora (9)

Nightly (10)

Firefox Mobile

7

8

9

Firefox Sync

Sync issues - The 503 storm of 2011
  • Fuel for the fire:
    • Instant Sync:
      • Created unexpected 30-50% spike in traffic
    • Broken Server-Sent backoff
    • Poor server capacity/health for existing users
  • Firefighting:
    • Infrastructure changes
      • DB updates
      • Load balancer updates
      • Webserver tuning
    • Migration
      • Added capacity
      • Pruned stale data
      • Migrate users from overloaded servers
  • Fireproofing:
    • Better modeling for current and expected traffic on Sync features
    • Ops review and sign off on feature pages
    • Required system-review for all client behavior changes
    • Require reliability metrics and dashboard of these on all channels
    • Instill a review of changes and metrics 1 week before merging on all channels
    • Fix backoff code in Fx8
    • Re-triage of maintenance bugs for any lurking issues
    • Blog posts coming from both Ops and Engineering

Add-on Builder

Add-on SDK

Release (1.1 -> Firefox 7)

Stabilization (1.2 -> Firefox 7, 8)

Development (1.3 -> Firefox 8, 9)

Feedback Summary

Desktop

Mobile

UX & User Research

Market Insights

Marketing, Press & Public Reaction

Questions, Comments, FYI

Actions this week

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox/Planning/2011-10-19&oldid=359448"