MozillaWiki

Test Pilot/2016Q3: Difference between revisions

< Test Pilot
(Created page with "=== 2016 Q3 Goals === * '''Objective:''' Get 75% of our users to try a new experiment within 24 hours of launch * '''Key Results:''' ** Push Notifications used by 80% of u...")
 
mNo edit summary
 
(19 intermediate revisions by 2 users not shown)
Line 1: Line 1:
=== 2016 Q3 Goals ===


* '''Objective:'''  Get 75% of our users to try a new experiment within 24 hours of launch
Test Pilot Q3 Objectives and Key Results
* '''Key Results:''' 
** Push Notifications used by 80% of users
** 5% of users share an experiment socially
** UI for new features lands


* '''Objective:''' Get 80% of users using the Test Pilot site and add-on in their first choice language
'''Final Score:  0.61'''
* '''Key Results:'''
** Site & add-on is available in FIGS languages
** Send one marketing email in FIGS languages


* '''Objective:'''  All new Test Pilot experiments support localization when appropriate
{| class="wikitable"
* '''Key Results:''' 
!
** Determine and document our supported solutions
!
** Site support for identifying experiments based on contextual information (eg. language, region, platform, etc)
! Final Status (Sep 30)
 
! Comments
* '''Objective:'''  Get 6% of our legacy users into the new Test Pilot program
|-
* '''Key Results:'''
! Objective 1
** Review & Implement the legacy migration program
! Grow the Test Pilot user base.
** Remove Firefox Accounts integration
! .72
 
!
* '''Objective:''' Graduate an experiment
|-
* '''Key Results:'''
| KR1
** '''TODO TBD''' (publicly digestible after action report, persist the add-on somewhere. write a sweet blog post about it afterwards explaining our reasoning and conclusions.) Adjust experiment page to show conclusions and link to AMO if it went there.
| Grow total Test Pilot user base by 66%
| <span style="color:green;font-weight:bold;">1.0</span>
| We started the quarter with around 2000 DAU and on Sep 26th we had 21120.
|-
| KR2
| Add localization support to Test Pilot and get 80% of users using Test Pilot in their first choice language.
| <span style="color:red;font-weight:bold;">0.0</span>
| We deprioritized this completely.
|-
| KR3
| All new Test Pilot experiments support localization when appropriate.
|  <span style="color:green;font-weight:bold;">1.0</span>
| We're marking this as a 1.0 because for our experiments it wasn't appropriate, because L10n was deprioritized (see KR2).
|-
| KR4
| Get 6% of our legacy users into the new Test Pilot program.
| <span style="color:orange;font-weight:bold;">0.6</span>
| The peak usage over the last 30 days before the migration started was 735932 users, making our goal 44156 users.  We ended up with 26254 new Test Pilot installs directly from this campaign, making for a 3.6% conversion.  There are still 170k people with the old add-on installed.
|-
| KR5
| Remove Firefox Accounts integration.
| <span style="color:green;font-weight:bold;">1.0</span>
|
|-
|
|
|
|
|-
! Objective 2
! Make it easy for existing Test Pilot users to learn about new experiments and start using them.
! .77
!
|-
| KR1
| Get new experiment within 24 hours of launch at a rate 10% higher than our baseline engagement ratio.
| <span style="color:green;font-weight:bold;">1.0</span>
| this should      have been "get x% of our users using a new experiment in y hours."  Since      the engagement ratio is based off of DAU and MAU it's not what we should      use in the first 24 hours. NoMore404s had 10k users in the first week.
|-
| KR2
| Implement a notification system in Test Pilot
| <span style="color:orange;font-weight:bold;">0.8</span>
| We did implement a push notification system, but it has some problems (weird empty messages) that we haven't solved and don't know how. We also added badging to our icon which has proven to be very successful for engagement (far more than the push notifications).
|-
| KR3
| 5% of active Test Pilot users share an experiment socially
| <span style="color:orange;font-weight:bold;">0.5</span>
| This is really hard to measure as people could just be copying/pasting links to their social media.  We've seen 363 "share" clicks from our site in the past 3 days, but weren't measuring earlier than that.  5% of our DAU (21120) is 1056 users.  Q3 had 1500 sessions starting from facebook, 635 from twitter.
|-
|
|
|
|
|-
! Objective 3
! Demonstrate that the Test Pilot experiment life cycle is valuable and sustainable.
! .35
!
|-
| KR1
| Ensure at least one experiment graduates
| <span style="color:red;font-weight:bold;">0.0</span>
|
|-
| KR2
| Ensure at least one new experiment is publicly available
| <span style="color:green;font-weight:bold;">4.0</span>
| We actually shipped 4 new experiments!  No More 404s, Tracking Protection, Min Vid, Page Shot (4.0 for good feelings, but 1.0 for actual score)
|-
| KR3
| Publish conclusions
| <span style="color:red;font-weight:bold;">0.0</span>
|
|-
| KR4
| Build an offboarding process
| <span style="color:orange;font-weight:bold;">0.38</span>
| This is being tracked in [https://github.com/mozilla/testpilot/milestone/19 Milestone TXP-18].  We've done 3/8 of the issues so far.
|}


=== Updates ===
* [https://mail.mozilla.org/pipermail/testpilot-dev/2016-August/000225.html Mid-Quarter check-in]
* Q3 Review was on Oct 7, 2016


=== Non-Goals ===
=== Non-Goals ===

Latest revision as of 17:52, 7 October 2016

Test Pilot Q3 Objectives and Key Results

Final Score: 0.61

Final Status (Sep 30) Comments
Objective 1 Grow the Test Pilot user base. .72
KR1 Grow total Test Pilot user base by 66% 1.0 We started the quarter with around 2000 DAU and on Sep 26th we had 21120.
KR2 Add localization support to Test Pilot and get 80% of users using Test Pilot in their first choice language. 0.0 We deprioritized this completely.
KR3 All new Test Pilot experiments support localization when appropriate. 1.0 We're marking this as a 1.0 because for our experiments it wasn't appropriate, because L10n was deprioritized (see KR2).
KR4 Get 6% of our legacy users into the new Test Pilot program. 0.6 The peak usage over the last 30 days before the migration started was 735932 users, making our goal 44156 users. We ended up with 26254 new Test Pilot installs directly from this campaign, making for a 3.6% conversion. There are still 170k people with the old add-on installed.
KR5 Remove Firefox Accounts integration. 1.0
Objective 2 Make it easy for existing Test Pilot users to learn about new experiments and start using them. .77
KR1 Get new experiment within 24 hours of launch at a rate 10% higher than our baseline engagement ratio. 1.0 this should have been "get x% of our users using a new experiment in y hours." Since the engagement ratio is based off of DAU and MAU it's not what we should use in the first 24 hours. NoMore404s had 10k users in the first week.
KR2 Implement a notification system in Test Pilot 0.8 We did implement a push notification system, but it has some problems (weird empty messages) that we haven't solved and don't know how. We also added badging to our icon which has proven to be very successful for engagement (far more than the push notifications).
KR3 5% of active Test Pilot users share an experiment socially 0.5 This is really hard to measure as people could just be copying/pasting links to their social media. We've seen 363 "share" clicks from our site in the past 3 days, but weren't measuring earlier than that. 5% of our DAU (21120) is 1056 users. Q3 had 1500 sessions starting from facebook, 635 from twitter.
Objective 3 Demonstrate that the Test Pilot experiment life cycle is valuable and sustainable. .35
KR1 Ensure at least one experiment graduates 0.0
KR2 Ensure at least one new experiment is publicly available 4.0 We actually shipped 4 new experiments! No More 404s, Tracking Protection, Min Vid, Page Shot (4.0 for good feelings, but 1.0 for actual score)
KR3 Publish conclusions 0.0
KR4 Build an offboarding process 0.38 This is being tracked in Milestone TXP-18. We've done 3/8 of the issues so far.

Updates

Non-Goals

The below were discussed as part of our goal setting but were not selected. They are included here for posterity and/or future planning.

  • Objective: Expose standardized experiment performance data in an easily consumable format
  • Key Results:
    • Build a standardized “report card” with high level statistics
    • Send an automated weekly summary email to executives
Retrieved from "https://wiki.allizom.org/index.php?title=Test_Pilot/2016Q3&oldid=1150617"