Leanplum Contextual Hints: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(First version for the document)
 
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Leanplum Contextual Hints Test Plan (iOS & Android) - July 2017 =
==Test Plan (iOS & Android) - July 2017 ==
; '''Goal:'''
; '''Goal:'''
With the initial implementation of the Leanplum SDK into our mobile browser products, we want to start A/B testing contextual hints to increase retention.
With the initial implementation of the Leanplum SDK into our mobile browser products, we want to start A/B testing contextual hints to increase retention.
Line 14: Line 14:
   
   
We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.  
We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.  
; '''iOS Hypothesis 3:'''
We believe that by promoting tips and tricks such as customizing settings to our users inside Firefox, we will increase DAUs and retention.
; '''iOS Hypothesis 4:'''
We believe that by promoting Pocket to our users inside Firefox, we will increase overall DAUs and retention of the Pocket app.
; '''iOS Hypothesis 5:'''
We believe that by promoting Pocket Stories to our users on Firefox, we will increase overall DAUs and retention of the Firefox browser.
; '''iOS Hypothesis 6:'''
We believe that introducing a new user to Firefox’s top features and get them using Firefox across devices will increase overall DAUs and retention of the Firefox browser.


{| class="wikitable"
{| class="wikitable"
|-
|-
! Hyp. # !! Message Name !! Trigger !! Deep Link  !! Message Type !! Event & Timing !! Targets !! Result
! Hyp. # !! Message Name !! Trigger !! Targets
|-
|-
| H1 || Clear Private Data - Focus Promotion || Clear private data || Focus Adjust link || Banner || Clear Private Data once  || Test: en-USPost-test: en, en-gb, de, zh-tw || N/A
| H1 || Opened New Tab - Focus Promotion || Open new tab || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw  
|-
|-
| H1 || Opened New Tab - Focus Promotion - Promote Default Browser || Open new tab || Focus Adjust link || Banner || Open new tab once || Test: en-USPost-test: en, en-gb, de, zh-tw || N/A
| H2 || Open Bookmark - Sync Promotion || Open bookmark || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|-
|-
| H2 || Open Bookmark - Sync Promotion || Open bookmark || firefox://open-url?deep-link=settings/fxa || Banner || Open bookmark once || Test: en-USPost-test: en, en-gb, de, zh-tw || N/A
| H2 || Save Bookmark - Sync Promotion || Save bookmark || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|-
|-
| H3 || Save Bookmark - Sync Promotion || Save bookmark || firefox://sign_up || Dialog/Banner || Save bookmark once  || Test: en-USPost-test: en, en-gb, de, zh-tw || N/A
| H2 || Save Login and Password - Sync Promotion || Save bookmark || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|-
| H3 || Search URL Area - Default Search Engine Promotion || Interacts with search url area || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|-
| H3 || Search URL Area - Homepage Setting Promotion || Interacts with search url area || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|-
| H3 || Open New Tab - New Tab Setting Promotion || Open new tab || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|-
| H4 || Open Bookmark - Pocket Promotion || Open bookmark || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|-
| H5 || Click on Pocket Stories - Pocket Promotion || Click on Pocket Stories || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|-
| H6 || Onboarding flow: Welcome slide -> FxAccount Sign up slide -> Search Settings banner -> Night Mode banner -> Bookmark banner -> Private browsing banner  || First run || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|}
|}


==Android==
==Android==
Line 40: Line 66:
; '''Android Hypothesis 3: '''
; '''Android Hypothesis 3: '''
We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.  
We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.  
; '''Android Hypothesis 4: '''
We believe that by promoting tips and tricks like the bookmark list or search engine settings to our users inside Firefox, we will increase DAUs and retention.
; '''Android Hypothesis 5: '''
We believe that by promoting Focus to our privacy conscious users inside Firefox, we will increase installs and DAUs in Focus, and increase retention on both apps (We know that iOS users who have Focus and Firefox installed retain better (Adjust)).
; '''Android Hypothesis 6:'''
We believe that introducing a new user to Firefox’s top features and get them using Firefox across devices will increase overall DAUs and retention of the Firefox browser.
; '''Android Hypothesis 7:'''
We believe by not promoting Pocket content to users who have opted out of Pocket in their Top Sites we are respecting the user choices and creating a better user experience.
; '''Android Hypothesis 8:'''
We believe by knowing when a user opens up the Firefox mobile app we can better measure the success of campaigns that encourage a user to open up the Firefox mobile app.


{| class="wikitable"
{| class="wikitable"
|-
|-
! Hyp. # !! Message Name !! Trigger !! Deep Link  !! Message Type !! Event & Timing !! Targets !! Result
! Hyp. # !! Message Name !! Trigger !! Targets  
|-
| H1 || Launch app - Default Browser promotion || App Start || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
|-
| H1 || Load Bookmark - Default Browser promotion || Load bookmark || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
|-
| H2 || Launch app - Add-ons promotion || App start || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw in US, CA, AU, UK, DE, TW, HK, SG
|-
| H5 || Open New Tab - Focus promotion || Open new tab || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
|-
| H4 || Save Bookmark - Bookmark List Promotion || Save bookmark || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
|-
| H1 || Log in to FxA - Default Browser Promotion || Log in to FxA || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
|-
|-
| H1 || Launch Promote - Default Browser || App Start || firefox://default_browser || Dialog || User starts app second time (only once)VsUser starts app third time (only once)One event a week or 2 weeks - ask Evan || Test: en-USPost-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG || N/A
| H1 || Add Bookmark 5 times - Default Browser Promotion || Add Bookmark 5 times || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG  
|-
|-
| H1 || Load Bookmark - Promote Default Browser || Load bookmark || firefox://default_browser || Dialog || Load Bookmark (1st time) oncevsLoad Bookmark (2 times) onceVsLoad Bookmark (5 times) once || Test: en-USPost-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG || N/A
| H3 || Save Bookmark - Sync Promotion || Save bookmark || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG  
|-
|-
| H2 || Firefox launches - add-ons promotion || App start || http://addons.mozilla.org || Dialog || User starts app 2nd time (only once)One event a week or 2 weeks || Test: en-US Post-test: en, en-gb, en-ca, de, zh-tw in US, CA, AU, UK, DE, TW, HK, SG || N/A
| H4 || Search URL Area - Default Search Engine Promotion || Interact with search url area || Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG  
|-
|-
| H3 || Save Password - Promote Sync || Save password || firefox://sign_up || Dialog/Banner || Save Password once, banner shows up the next time you launch app || Test: en-USPost-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG || N/A
| H6 || Onboarding flow: Original default onboarding slides -> Reader mode banner -> Bookmark banner -> Default browser banner  || First run || Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
|}
|}
For more detail about the triggers and deep links in Android, please go to this page[http://gecko.readthedocs.io/en/latest/mobile/android/fennec/mma.html]

Latest revision as of 22:04, 16 April 2018

Test Plan (iOS & Android) - July 2017

Goal:

With the initial implementation of the Leanplum SDK into our mobile browser products, we want to start A/B testing contextual hints to increase retention.

Overarching Hypothesis:

We believe that by promoting contextual hints that helps users explore and change settings within the browser, we will drive DAUs and retention.

iOS

iOS Hypothesis 1:

We believe that by promoting Focus to our privacy conscious users inside Firefox, we will increase installs and DAUs in Focus, and increase retention on both apps (We know that users who have Focus and Firefox installed retain better (Adjust)).

iOS Hypothesis 2:

We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.

iOS Hypothesis 3:

We believe that by promoting tips and tricks such as customizing settings to our users inside Firefox, we will increase DAUs and retention.

iOS Hypothesis 4:

We believe that by promoting Pocket to our users inside Firefox, we will increase overall DAUs and retention of the Pocket app.

iOS Hypothesis 5:

We believe that by promoting Pocket Stories to our users on Firefox, we will increase overall DAUs and retention of the Firefox browser.

iOS Hypothesis 6:

We believe that introducing a new user to Firefox’s top features and get them using Firefox across devices will increase overall DAUs and retention of the Firefox browser.

Hyp. # Message Name Trigger Targets
H1 Opened New Tab - Focus Promotion Open new tab Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H2 Open Bookmark - Sync Promotion Open bookmark Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H2 Save Bookmark - Sync Promotion Save bookmark Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H2 Save Login and Password - Sync Promotion Save bookmark Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H3 Search URL Area - Default Search Engine Promotion Interacts with search url area Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H3 Search URL Area - Homepage Setting Promotion Interacts with search url area Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H3 Open New Tab - New Tab Setting Promotion Open new tab Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H4 Open Bookmark - Pocket Promotion Open bookmark Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H5 Click on Pocket Stories - Pocket Promotion Click on Pocket Stories Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw
H6 Onboarding flow: Welcome slide -> FxAccount Sign up slide -> Search Settings banner -> Night Mode banner -> Bookmark banner -> Private browsing banner First run Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw

Android

Android Hypothesis 1:

We believe that by prompting users to change their default browser to Firefox, we will increase DAUs and retention.

Android Hypothesis 2:

We believe that by promoting add-ons to our users inside Firefox, we will increase DAUs and retention.

Android Hypothesis 3:

We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.

Android Hypothesis 4:

We believe that by promoting tips and tricks like the bookmark list or search engine settings to our users inside Firefox, we will increase DAUs and retention.

Android Hypothesis 5:

We believe that by promoting Focus to our privacy conscious users inside Firefox, we will increase installs and DAUs in Focus, and increase retention on both apps (We know that iOS users who have Focus and Firefox installed retain better (Adjust)).

Android Hypothesis 6:

We believe that introducing a new user to Firefox’s top features and get them using Firefox across devices will increase overall DAUs and retention of the Firefox browser.

Android Hypothesis 7:

We believe by not promoting Pocket content to users who have opted out of Pocket in their Top Sites we are respecting the user choices and creating a better user experience.

Android Hypothesis 8:

We believe by knowing when a user opens up the Firefox mobile app we can better measure the success of campaigns that encourage a user to open up the Firefox mobile app.

Hyp. # Message Name Trigger Targets
H1 Launch app - Default Browser promotion App Start Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H1 Load Bookmark - Default Browser promotion Load bookmark Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H2 Launch app - Add-ons promotion App start Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw in US, CA, AU, UK, DE, TW, HK, SG
H5 Open New Tab - Focus promotion Open new tab Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H4 Save Bookmark - Bookmark List Promotion Save bookmark Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H1 Log in to FxA - Default Browser Promotion Log in to FxA Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H1 Add Bookmark 5 times - Default Browser Promotion Add Bookmark 5 times Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H3 Save Bookmark - Sync Promotion Save bookmark Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H4 Search URL Area - Default Search Engine Promotion Interact with search url area Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG
H6 Onboarding flow: Original default onboarding slides -> Reader mode banner -> Bookmark banner -> Default browser banner First run Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw

For more detail about the triggers and deep links in Android, please go to this page[1]

Retrieved from "https://wiki.allizom.org/index.php?title=Leanplum_Contextual_Hints&oldid=1192400"