Webdev:Meetings:2009-09-01

Open Items

• AMO team is seeking ideas: We generate CSVs from our statistics data for add-on authors. There are 3 date groupings and 8 different ways to plot the data. The problem is, the historical data continues to grow and we're running out of memory building these huge CSVs on the fly. Ideas:
  • When the uninstall survey started dying on CSVs we used a cron to build them and cache them to disk. If we do this for all our add-ons that's well over 200,000 files and growing. Perhaps we can combine this with one of the other ideas.
  • Provide less historical data. Right now it goes all the way back. Restricting that is weak sauce.
  • Reduce the number of groupings/plots. What if we just provided CSVs for daily downloads with a couple sets of columns. That's only ~15000 files per set of columns. Still a lot.
    • 1 row = 1 day, right? What if past $x weeks in history, we only offered monthly totals? i.e. data older than 6 months is 1 row = 1 (week/month)
  • Generate CSVs for add-ons with more than $x weeks of history. Eventually we'll have #1.
  • Write something way lighter weight to build CSVs on the fly. We can't scale this way forever though.
  • Limit the number of rows returned but provide paging params to view older ranges of data
  • Output CSV as it is generated and bypass Cake views, thus avoiding the need to generate huge arrays of data
  • add more ideas! thx