Services/Meetings/2011-09-20: Difference between revisions

Services/Meetings/2011-09-20 (view source)

Revision as of 07:45, 20 September 2011

2,027 bytes added , 20 September 2011

→‎MySQL upgrades

Rsoderberg

Confirmed users

96

edits

@@ Line 15: / Line 15: @@
 == MySQL upgrades ==
+Upgrading mysql in production starting today (Tuesday, Sep 20).  Over the next ten days, we plan to upgrade all 130 servers.  This is a Q2 goal deferred to Q3, and is also expected to resolve the ongoing SCL2 crash issues.
+Currently we're running InnoDB databases using MySQL 5.1.44, build maria75, dated March 2010.  We're upgrading to MySQL 5.1.58, build percona12.9, dated September 2011.
+The new version passes load tests and will be deployed gradually in case something happens.
+User data remains intact during the upgrade.  The desired user experience is 10 minutes of 503 Server Error.  As currently implemented, 503 shows a Server Issue error bar to users.  This will be more of that.  Firefox 7+ improve the 503 experience, ask Philikon or Marina for details.
+Server responsiveness may fluctuate for the first few hours as MySQL adapts to what we've done to it.  We saw an interesting pattern over several hours as it calibrates to the write performance of the server.
+The approximate timeline is:
+* Tue Sep 20 - Upgrade 30 database servers.
+* Thu Sep 22 - Upgrade 50 database servers.
+* Tue Sep 27 .. Thu Sep 29 - Upgrade the rest.
+We've discussed user notification a bit tonight on IRC, but no final decision has been made. I expect there will be a single tweet from the Mozilla Services account.
+There are three classes of database hardware: 6-drive iX (old PHX), 24-drive iX (new PHX), 9-drive Cuil (SCL2), with a variety of Linux md layouts.
+Some of each will be upgraded each time to improve our chances of catching any issues.  Rolling back will either take a few minutes – same as the upgrade – or require a user migration.
+We have a variety of pencil graphs tracking Sync, MySQL, InnoDB statistics on all production databases.  There are many days of historical data.
+Upgrading any single database will likely require a user-facing downtime of about 10 minutes. On any single server, MySQL may choose to make us wait up to 60 minutes for the mandatory clean shutdown.  Hopefully this is rare.
 == Membase ==