Confirmed users
632
edits
Loop/Load Handling: Difference between revisions
→Service Soft Start
| Line 6: | Line 6: | ||
This mechanism allows Mozilla to gradually ramp up system load after the feature makes it into release. In a nutshell: | This mechanism allows Mozilla to gradually ramp up system load after the feature makes it into release. In a nutshell: | ||
By default, release | By default, Firefox 34 release and beta clients will have two prefs set to "true": loop.enabled and loop.throttled. Upon '''''first''''' startup, each client will select a random number in the range of 1 to 2<sup>24</sup>-2 and write it into the "loop.soft_start_ticket_number" pref. Then, upon this and every subsequent startup, each client will check the value of the "loop.throttled" pref. If set to true, then the client checks the value of a DNS A record (tentatively "soft-start.loop.services.mozilla.com" -- see {{Bug|1060809}}), which is required to be in the range 127.0.0.0 - 127.255.255.255. If the record is outside this range, or if there is an error retrieving the A record, then the client does not activate the Loop feature. | ||
If the record is successfully retrieved, then the low 24 bits of the address are treated as a "now serving" number, and compared to the value stored in "loop.soft_start_ticket_number". If the value is strictly greater than the selected ticket number, then the feature is activated, and the "loop.throttled" pref is set to false (which will bypass this procedure for all subsequent startups). | If the record is successfully retrieved, then the low 24 bits of the address are treated as a "now serving" number, and compared to the value stored in "loop.soft_start_ticket_number". If the value is strictly greater than the selected ticket number, then the feature is activated, and the "loop.throttled" pref is set to false (which will bypass this procedure for all subsequent startups). | ||
| Line 67: | Line 67: | ||
''I'm a crusty old perl programmer, so the suggestion I have for generating the IP address for an arbitrary load value looks like this; I'm sure there are more elegant solutions in other languages, perl being what it is. Simply replace the "50" at the end with the load level you'd like to get an IP address for:'' | ''I'm a crusty old perl programmer, so the suggestion I have for generating the IP address for an arbitrary load value looks like this; I'm sure there are more elegant solutions in other languages, perl being what it is. Simply replace the "50" at the end with the load level you'd like to get an IP address for:'' | ||
perl -e 'print join(".",unpack("C*",pack("N",127<<24|int(((1<<24)-1)*(shift)/100))))."\n"' 50 | perl -e 'print join(".",unpack("C*",pack("N",127<<24|int(((1<<24)-1)*(shift)/100))))."\n"' 50 | ||
This process repeats for Firefox 35, although instead of activating the feature, the throttle will be used to move the button from being in the customization palette by default to being in the toolbar by default. | |||
=== Recommended Handling === | |||
'''October 13''' - While Firefox 34 is in Beta, we probably want to set the throttle to 100%, so that it is available to all beta users. Early beta of 33 had the feature active and in the default toolbar set, so we already have a good idea that we can handle the resulting load. | |||
'''November 19''' - As we approach the uplift of 34 to release, we want to reduce the throttle to a lower value, around 10%. I propose that we do this several days out, to make sure we don't run into DNS caching issues. | |||
'''November 19 - December 30''' - Gradually increase the throttle. Observe server load and behavior (Loop server, Simple Push server, and TokBox infrastructure). As we are convinced the servers are equipped to handle the load, we increase the throttle according to how much headroom we believe the servers have. I would recommend giving at least 48 hours for the load to settle before adjusting the throttle value. During this time period, release users will gain access to the feature, and beta users will have the button appear in their default toolbar. | |||
'''December 30''' - Leading up to the uplift of 35 to release, we again reduce the throttle to 10%. Note that the throttle acts as a ratchet: once active, the service remains active. Reducing the throttle will not cause the feature to disappear from 34 browsers. | |||
'''December 30 - February 15''' - Repeat the throttle increase process described above. | |||
'''February 15''' - {{Bug|1073218}} should land in Firefox 36, which will be in release around this date. We still want to give people time to upgrade before leaving users on 35 without a button in the default toolbar; one release cycle should be long enough. | |||
'''March 30''' - Remove the throttle DNS record. | |||
== Simple Push Server Load Distribution == | == Simple Push Server Load Distribution == | ||