Funnelcake: Difference between revisions

== Firefox Funnelcakes ==

== Anatomy of a successful Firefox install ==

Funnelcakes change how our download and install funnel works. Here’s how things normally go:

# A user decides to download Firefox and finds their way to the Mozilla web site and the Firefox download page hosted there.

# The user clicks on the “Download Firefox” button. Server technology (“the bouncer”) determines which platform, locale, channel and version the user should get, and redirects the user to that particular binary file on our CDN. The download begins.  

# When the download has finished, the user installs Firefox using the downloaded file.  

# Next, the user opens Firefox. Since this is the first time the user has opened a fresh install of Firefox, Firefox will automatically request a page served by our website, the “First Run” page.

# The user enjoys and depends on Firefox for many years. It helps them navigate through all the times of their life.

When a funnelcake experiment is underway, the above steps are different:

* Some subset of users at step 2 download a special version of Firefox (the funnelcake version) with a unique anonymous number associated with the entire experiment (funnelcake id).

* When those funnelcake users visit the First Run page in step 4, the URL for that page includes a parameter (usually ?f=funnelcake-id) identifying them as part of the cohort. This allows us to understand how the funnelcake affected install rates.

== How to run a Funnelcake experiment ==

'''TLDR;''' Design an experiment, ask the Release Engineering team to build a special version of Firefox, and ask the Mozilla.org team to distribute that special version to a particular segment of downloaders.

'''More specifically:'''

# At least 4 weeks^* before you hope to launch your experiment, design it. A good experiment will have a hypothesis, a measurement method, a timeline, and any details someone else might require to reconstruct it at a later date. The Firefox Growth team has experiment templates for [https://docs.google.com/document/d/1iGaLTwPYA6AmxJ8EssuMyNoErsgun4vPOcC-ijfixIY/edit Google Docs] and [https://mana.mozilla.org/wiki/display/FIREFOX/BLANK_Template_AB_Test_Details Mana]. Here is an [https://mana.mozilla.org/wiki/display/FIREFOX/Vietnam+Funnelcake+with+AVIM+Pre-Installed+V2 example funnelcake experiment]. You can share your experiment by adding it to this [https://mana.mozilla.org/wiki/display/FIREFOX/Detailed+Experiment+Reports+of+Firefox+Growth+Team list of experiments].

# Convince stakeholders on teams involved that the experiment is worthwhile. Talk to the Growth Team if you want help assessing this.

# At least 2 weeks^* before you hope to launch your experiment,

## File a bug for the release engineering team explaining what the funnelcake should include, which version and channel of Firefox it should be based on, what locale it is for, and when distribution should begin. [https://bugzilla.mozilla.org/show_bug.cgi?id=1257899 Here is an example RE bug].

## Add your funnelcake to [https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0 the big list of all funnelcakes ever] and fill in all the fields that spreadsheet requires. This is also where you’ll get your funnelcake number.

## File a bug for the Mozilla.org team to build an Optimizely experiment to distribute the funnelcake to a specific segment (include relevant factors such as locale, geography, user agent, segment size), as well as any other changes. [https://bugzilla.mozilla.org/show_bug.cgi?id=1202004 Here is an example Mozilla.org bug].

 

 

 

A: It depends on your needs, but a common rule of thumb is you need 40,000-50,000 installs (which implies a larger number of downloads -- determined by the conversion rate of the segment you’re targeting). Use a tool like [https://www.optimizely.com/resources/sample-size-calculator/?conversion=1&effect=1&significance=95 Optimizely’s sample size calculator] to plan your experiment.

 

A: [https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0 Here is a list of all the funnelcakes ever run]!

 

A: They are arbitrary (they don’t have any algorithmic meaning), but sequential. You can see the sequence on [https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0 the big list of funnelcakes].

 

'''Q: How do we measure retention using funnelcakes?'''

A: One field in the big list of funnelcakes is “distribution ID”. This field is included in the channel that Firefox sends to telemetry. We can monitor telemetry data (including retention) for users of a particular channel (such as a funnelcake) using [https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0 tools like this].

 

A: The firefox user advocacy and research teams can target specific distribution IDs (such as the one assigned to a funnelcake) with [[Advocacy/heartbeat|heartbeat surveys]]. Furthermore, we can distribute funnelcakes to [https://www.usertesting.com/ usertesting.com] testers.