Confirmed users, Bureaucrats and Sysops emeriti
1,737
edits
Test Pilot/Metrics: Difference between revisions
m
no edit summary
mNo edit summary |
mNo edit summary |
||
| Line 6: | Line 6: | ||
* It should provide an actionable data-point to help you improve your test. | * It should provide an actionable data-point to help you improve your test. | ||
* It should provide product owners and the Test Pilot Council with the data necessary to make informed decisions about the outcome of each test. For example: should this test result in product shipping in Release or AMO? Should this test be used as a springboard for future UX iteration? | * It should provide product owners and the Test Pilot Council with the data necessary to make informed decisions about the outcome of each test. For example: should this test result in the product shipping in Release or AMO? Should this test be used as a springboard for future UX iteration? | ||
== Core and Secondary Metrics == | == Core and Secondary Metrics == | ||
| Line 20: | Line 20: | ||
=== Usage Rate per User === | === Usage Rate per User === | ||
'''Answers the Question:''' Is this type of experience people are looking for? | '''Answers the Question:''' Is this the type of experience people are looking for? | ||
'''How we measure it:''' Measuring usage across different tests is challenging since a healthy usage rate on one service might not look like a healthy usage rate on another. To measure usage for your idea, the Test Pilot Council will determine a single event in each usage cycle of your test to be captured (eg. for a screenshotting test, the key usage event might be fired when a screenshot is taken). Usage will be defined as the rate this event is fired over time. Usage is recorded from the time a user starts | '''How we measure it:''' Measuring usage across different tests is challenging since a healthy usage rate on one service might not look like a healthy usage rate on another. To measure usage for your idea, the Test Pilot Council will help you determine a single event in each usage cycle of your test to be captured (eg. for a screenshotting test, the key usage event might be fired when a screenshot is taken). Usage will be defined as the rate this event is fired over time. Usage is recorded from the time a user starts a test until the test concludes, so that if a user quits a test, this will negatively impact aggregate usage rate. | ||
| Line 29: | Line 29: | ||
{| class="wikitable" | {| class="wikitable" | ||
! | |||
! Low NPS | ! Low NPS | ||
! High NPS | ! High NPS | ||
|- | |- | ||
! High Usage | ! High Usage | ||
|This scenario likely indicates you’re building something users want to like, but the quality and polish of your experience is lacking. | |This scenario likely indicates you’re building something users want to like, but the quality and polish of your experience is lacking. Recommendations: | ||
* Focus of quality of experience issues. | |||
* Dive into feedback with a specific focus on reported bugs and other gripes. | |||
* Engage with the UX team for directed feedback about your test. | |||
| Congrats, your idea is awesome and you are awesome. Let’s ship this thing! | | Congrats, your idea is awesome and you are awesome. Let’s ship this thing! | ||
|- | |- | ||
! Low Usage | ! Low Usage | ||
| The good news is that there are a lot of options to try out here. | | The good news is that there are a lot of options to try out here. Start by looking at retention metrics. Are people joining and leaving your test, or do they simply not join at all? <br> In the former case, dive into user feedback to determine issues with build quality and feature set. In the latter case, start by engaging with the Product team to help repackage your test to make it more intelligible/appealing to potential users. | ||
| This scenario likely means you’ve got a high polish product, but that it may not be addressing broad user problems. | | This scenario likely means you’ve got a high polish product, but that it may not be addressing broad user problems. Recommendations: | ||
Focus | * Focus on feedback addressing feature requests. | ||
* Engage with the Product team to make sure you are describing your feature accurately to potential users. | |||
* Consider simplifying your offering or making it more accessible. | |||
* Consider shipping in AMO. | |||
|} | |} | ||
| Line 58: | Line 65: | ||
'''Answers the question:''' Is this idea sticky? Does it work for users? | '''Answers the question:''' Is this idea sticky? Does it work for users? | ||
'''How we measure it:''' Retention is | '''How we measure it:''' Retention is the % of users who keep use an idea over time and Churn is the rate at which users stop using. | ||
'''How to deal with low retention & high churn:''' Dive into feedback. Is your idea getting in people’s way? Is build quality an issue? Retention may be high if your idea sucks but is unintrusive. On the other hand, retention may be very low if your idea is pretty good but constantly barking at people or interrupting their primary task. If low retention correlates to high usage, this is likely the case. Use retention and churn to understand how the experience of your idea fits into users workflows. | '''How to deal with low retention & high churn:''' Dive into feedback. Is your idea getting in people’s way? Is build quality an issue? Retention may be high if your idea sucks but is unintrusive. On the other hand, retention may be very low if your idea is pretty good but constantly barking at people or interrupting their primary task. If low retention correlates to high usage, this is likely the case. Use retention and churn to understand how the experience of your idea fits into users workflows. | ||
| Line 81: | Line 88: | ||
'''How we measure it:''' When users leave | '''How we measure it:''' When users leave a test, we ask them to provide a reason from a list of four (“This thing is broken”, “I don’t like this feature”, “This isn’t useful for me”, Something else”). Users engaged in a test can provide feedback any time they like (“Something seems broken”, “Request a feature”, “This is cool”, “Something else”). Users can optionally provide a more detailed description for each. | ||
'''Answers the question:''' Why do users report they are leaving my test? | '''Answers the question:''' Why do users report they are leaving my test? | ||
'''What to do with this data:''' Use it to help you make informed decisions about high-impact changes to make to your test. | '''What to do with this data:''' Use it to help you make informed decisions about high-impact changes to make to your test. | ||
| Line 92: | Line 99: | ||
'''How do we measure it:''' | '''How do we measure it:''' | ||
A | A user bounces if they come to your test details page without the test installed and then leave the page without installing your test. | ||
=== Adoption Curve === | === Adoption Curve === | ||
'''Answers the question:''' Do specific events in Test Pilot (promotions etc) or changes to the packaging of my | '''Answers the question:''' Do specific events in Test Pilot (promotions, etc) or changes to the packaging of my add-on move the needle? This metric is useful to the Test Pilot team to help us understand how to better draw users to your idea (and to future ideas). | ||
=General for Test Pilot= | =General for Test Pilot= | ||
== Core Metrics == | == Core Metrics == | ||
=== % of Users Engaged in 0,1,2,3...n tests over time === | === % of Users Engaged in 0,1,2,3...n tests over time === | ||
'''Answers the question:''' Does Test Pilot promote engagement? Are the current batch of ideas useful overall? Are some ideas more useful than others? | |||
=== Average NPS response score across active tests === | === Average NPS response score across active tests === | ||
'''Answers the question:''' Does Test Pilot make Firefox a more recommendable product? Are the current batch of ideas attractive overall? | |||
=== Health Metrics === | |||
'''Answers the question:''' How is the site code and infrastructure performing? This is standard stuff for a website, eg. response times, percentage of errors, etc. | |||
== Secondary Metrics == | == Secondary Metrics == | ||
=== Average Idea Duration === | === Average Idea Duration === | ||
'''Answers the question:''' How long should ideas plan to be in Test Pilot? | |||
=== Overall NPS response rate across active tests === | === Overall NPS response rate across active tests === | ||
'''Answers the question:''' Do the current batch of tests promote engagement? | |||
=== Average Engaged Retention & Churn === | === Average Engaged Retention & Churn === | ||
'''Answers the question:''' Are current tests working for users? | |||
'''How we measure it:''' This is aggregate churn data of all active tests. | |||
=== Retention & Churn === | === Retention & Churn === | ||
'''Answers the question:''' Is Test Pilot working for users? Is the service itself healthy. | |||
''' | '''How we measure it''': We define a churned user in this case as a user who has uninstalled Test Pilot or who has not interacted with Test Pilot UI in one month. | ||
=== Bounce Rate through Sign Up and Install === | === Bounce Rate through Sign Up and Install === | ||
'''Answers the question:''' Is the Test Pilot flow effective? | |||
=== MAUs v. Total Sign Ups === | === MAUs v. Total Sign Ups === | ||
This may be a vanity metric, but it will help give us a sense of the scale of our population and give us insight into the statistical significance of our data. | This may be a vanity metric, but it will help give us a sense of the scale of our population and give us insight into the statistical significance of our data. | ||
=== Average number of ideas installed === | === Average number of ideas installed === | ||
Detail concerning engagement. May not have a clear value from launch, except that it might help us define user typologies down the road, and gives a bit more insight into the ways in which users are making use of Test Pilot. | Detail concerning engagement. May not have a clear value from launch, except that it might help us define user typologies down the road, and gives a bit more insight into the ways in which users are making use of Test Pilot. | ||