Firefox/Accounts/Progress Report/2017Q3: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Update 1.1 with most recent issues)
(completed updates for objective 3)
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
__NOTOC__
__NOTOC__
<big><big>'''August 8, 2017'''</big></big>
<big><big>'''September 25, 2017'''</big></big>


== Q3 OKR Progress ==
== Q3 OKR Progress ==
Line 17: Line 17:
|style="text-align: center; background-color: #f9d9a8"|Aug 8
|style="text-align: center; background-color: #f9d9a8"|Aug 8
|style="text-align: center; background-color: #f9d9a8"|Aug 21  
|style="text-align: center; background-color: #f9d9a8"|Aug 21  
|style="text-align: center; background-color: #f9d9a8"|Sep 4
|style="text-align: center; background-color: #f9d9a8"|Sep 5
|style="text-align: center; background-color: #f9d9a8"|Sep 18
|style="text-align: center; background-color: #f9d9a8"|Sep 18
|style="text-align: center; background-color: #f9d9a8"|Score
|style="text-align: center; background-color: #f9d9a8"|Score
Line 26: Line 26:
|align="center"|65%
|align="center"|65%
|align="center"|70%
|align="center"|70%
|align="center"|
|align="center"|75%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|100%
|align="left"|
|align="center"|1.0
|
|
Priorities for support of Quantum/Photon in 57:
*[https://waffle.io/mozilla/fxa?milestone=FxA-130:%20Crouton%20Flow Crouton Flow Issues for 57 release.]
*[https://waffle.io/mozilla/fxa?milestone=FxA-130:%20Crouton%20Flow Crouton Flow Issues for 57 release.]
Active in this sprint:
*[https://github.com/mozilla/fxa-auth-server/issues/1285 Issue 1285: Device names with non-BMP characters are rejected.]
*[https://github.com/mozilla/fxa-content-server/issues/4193 Issue 4193: Users who verify their email while at the CWTS screen may never start syncing.]
*[https://github.com/mozilla/fxa-content-server/issues/5037 Issue 4094: Bounce not triggering from Choose What To Sync screen]
*[https://github.com/mozilla/fxa-auth-db-mysql/issues/229 Issue 229: Update device name column to utf8mb4]
|
|
*Making good progress. Have 37% remaining.
*Done!
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-130:%20Crouton%20Flow Crouton Flow Burndown Chart]
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-130:%20Crouton%20Flow Crouton Flow Burndown Chart]
|-
|-
Line 45: Line 41:
|align="center"|50%
|align="center"|50%
|align="center"|75%
|align="center"|75%
|align="center"|
|align="center"|75%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|100%
|align="left"|
|align="center"|1.0
|
|
Quality Priorities for Q3:
*[https://waffle.io/mozilla/fxa?milestone=FxA-131:%20Q3%20 Quality related P1s that we’ve identified to fix in Q3.]
*[https://waffle.io/mozilla/fxa?milestone=FxA-131:%20Q3%20 Quality related P1s that we’ve identified to fix in Q3.]
Active in this sprint:
*[https://github.com/mozilla/fxa-content-server/issues/5193 Issue 5193: Sign-up flow: Add a “confirm password” field to minimize errors entering password, minimizing password reset.]
*[https://github.com/mozilla/fxa-content-server/issues/4863 Issue 4863: Add descriptions to Choose What to Sync]
*[https://github.com/mozilla/fxa-content-server/issues/5194 Issue 5194: Sign-up/in flow: Ask for email first]
*[https://github.com/mozilla/fxa-content-server/issues/5198 Issue 5198: Settings: Communication preferences can end up in an odd state]
|
|
*Initial confidence assessment wasn't based on much data
*Done!
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-131:%20Q3%20Quality Quality P1s Burndown] shows we’re on the ideal line
*Completed 13 out of 22, or 59%.
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-131:%20Q3%20Quality Quality P1s Burndown]
|-
|-
|align="left"| 1.3 Increase by 15% the rate of users that become multi-device as a result of going through our first run flow.
|align="left"| 1.3 Increase by 15% the rate of users that become multi-device as a result of going through our first run flow.
Line 64: Line 57:
|align="center"|50%
|align="center"|50%
|align="center"|50%
|align="center"|50%
|align="center"|
|align="center"|40%
|align="center"|
|align="center"|0%
|align="center"|
|align="center"|0%
|align="center"|
|align="center"|0.0
|
|
The onboarding team, marketing retention team and Mozilla.org team would like our help to improve user retention (through converting first run users to multi-device users).
*Supports the onboarding team, marketing retention team and Mozilla.org team to improve user retention (through converting first run users to multi-device users).
*[https://sql.telemetry.mozilla.org/dashboard/fxa-q3-okr-1-3 Dashboards related to user retention]
*[https://sql.telemetry.mozilla.org/dashboard/fxa-q3-okr-1-3 Dashboards related to user retention]
*[https://github.com/mozilla/fxa-content-server/pull/5332 Issue 5332: ABC test for CAD phase 3 (deep link)]
|
|
*We've seen an increase in views but no noticeable increase in what happens after.
*Didn't achieve this. Had a lot of questions about the metrics.
*Need to start shipping some of the quality improvements before we change the confidence level.
|-
|-
|align="left"| 1.4 Fix newsletter subscription when email is confirmed on a 2nd device.
|align="left"| 1.4 Fix newsletter subscription when email is confirmed on a 2nd device.
Line 79: Line 72:
|align="center"|95%
|align="center"|95%
|align="center"|95%
|align="center"|95%
|align="center"|
|align="center"|95%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|1.0
|
|
Fix the problem of how we currently lose newsletter subscribers if users confirm their emails on a different device.
Fixes a problem with getting marketing newsletter opt-in data over to Salesforce.
|
|
*Merged and should ship - we have everything but the measuring.
*Done!
*Will help marketing’s metrics.
|-
|-
|rowspan=2| 2. To have improved the quality of the Firefox Account experience on mobile.
|rowspan=2| 2. To have improved the quality of the Firefox Account experience on mobile.
   
   
| align="left"|2.1 Measure and A/B test best push opt-in screen on iOS.  
| align="left"|2.1 Measure and A/B test best push opt-in screen on iOS.  
|align="center"|50%
|align="center"|50%
|align="center"|40%
|align="center"|40%
|align="center"|55%
|align="center"|55%
|align="center"|
|align="center"|55%
|align="center"|
|align="center"|50%
|align="center"|
|align="center"|50%
|align="center"|
|align="center"|0.7
|align="left"|
|align="left"|
*[https://docs.google.com/spreadsheets/d/1tPE4KSNd_RU-ahSAK7a5hMcP5EAKGMlHJHegna87xLg/edit#gid=0 FxA Sync prioritization backlog for iOS]
*[https://docs.google.com/spreadsheets/d/1tPE4KSNd_RU-ahSAK7a5hMcP5EAKGMlHJHegna87xLg/edit#gid=0 FxA Sync prioritization backlog for iOS]
*[https://mozilla.invisionapp.com/share/76CPM7AC2#/screens/240561291_Desktop-_Connect_Another_Device Proposed mock up]
|
|
*Rfeely shared mock ups on how it should look. Ryan prefers it to be native. Not sure of impact of this.
*PR reviewed but not merged.
**[https://mozilla.invisionapp.com/share/76CPM7AC2#/screens/240561291_Desktop-_Connect_Another_Device Proposed mock up]
*Review brought up some interesting points to address.
*We don’t think iOS app. itself has something built in to perform the A-B test. Might be best to make it web based and use our own tools.
*Feature doc is in draft.  
|-
|-
| align="left"|2.2 Complete 80% of Q3 mobile bugs.
| align="left"|2.2 Complete 80% of Q3 mobile bugs.
Line 112: Line 102:
|align="center"|60%
|align="center"|60%
|align="center"|60%
|align="center"|60%
|align="center"|
|align="center"|60%
|align="center"|
|align="center"|65%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|0.8
|align="left"|  
|align="left"|  
Identification and fixing of the most impactful bugs on iOS
Identification and fixing of the most impactful bugs on iOS  
*[https://waffle.io/mozilla/fxa?milestone=FxA-132:%20Q3%20Mobile Mobile Bugs for Q3]
|
|
*Came up with a list of 13 bugs that were reviewed with Stefan
*8 out of 13 are done - 80% of 80% done as of 9/25
*10 of the 13 we have high confidence in completing
*[https://waffle.io/mozilla/fxa/metrics/burndown?milestone=FxA-132:%20Q3%20Mobile Mobile Quality Burndown Chart]
|-
|-
|rowspan=4| 3. To have completed all features that are still in flight from Q1 and Q2.
|rowspan=4| 3. To have completed all features that are still in flight from Q1 and Q2.
Line 128: Line 117:
|align="center"|75%
|align="center"|75%
|align="center"|50%  
|align="center"|50%  
|align="center"|
|align="center"|25%
|align="center"|
|align="center"|0%
|align="center"|
|align="center"|0%
|align="center"|
|align="center"|0.3
|align="left"|
|align="left"|
Allow users to test out device pairing. This does not necessarily mean that QR codes are used.
*Allow users to test out device pairing. This does not necessarily mean that QR codes are used.
*Feature doc and Issues coming soon.
|  
|
*Pushed forward the challenge with auto-focus.
*Phil and Shane to get together to chat about a feature doc for this
*There are things we got done this quarter that we can roll forward.
*Waiting for August 8th, when Android will support deeplinking in 55.
*Great analysis.
*Test plan updated.
*Plan is to enable it in train-93 because 55 will be out at that point.
*We need to think about anything we need to get into 56 beta.
|-
|-
| align="left"|3.2 Ship secondary email.
| align="left"|3.2 Ship secondary email.
Line 146: Line 132:
|align="center"|80%
|align="center"|80%
|align="center"|90%
|align="center"|90%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|0.9
|
|
We are allowing users to add a secondary email. Allows users to change primary email.
*We are allowing users to add a secondary email. Allows users to change primary email.
|
|
*Only thing remaining is how clients would handle the change on their GUI/interfaces
*Caching bug fixed
*Can disable or enable on the content server.
*Client code is in
*Works for AMO, and they have an issue to fix one little thing
*Pocket was the only remaining thing.
|-
|-
| align="left"|3.3 Land support for relier encryption keys.  
| align="left"|3.3 Land support for relier encryption keys.  
Line 160: Line 148:
|align="center"|70%
|align="center"|70%
|align="center"|75%
|align="center"|75%
|align="center"|
|align="center"|75%
|align="center"|
|align="center"|75%
|align="center"|
|align="center"|80%
|align="center"|
|align="center"|0.9
|align="left"|
|align="left"|
Relier encryption keys  
Relier encryption keys  
|
|
*Did some work on this but not enough to change confidence further.
*Have done a lot of good work on it but it’s not totally closed out.  
*Biggest risk is not getting the crypto review done soon enough (reliance on other teams).  
*Still working on the reviews.
*Currently under scrutiny by two crypto specialists.
*Can’t ship until we clear the security reviews.
*[https://github.com/mozilla/fxa/issues/245 Issue 245: Prototype key support api for Firefox Notes]
|-
|-
| align="left"|3.4 Integrate 100% of high priority Amplitude events.  
| align="left"|3.4 Integrate 100% of high priority Amplitude events.  
Line 174: Line 165:
|align="center"|75%
|align="center"|75%
|align="center"|75%
|align="center"|75%
|align="center"|
|align="center"|75%
|align="center"|
|align="center"|75%
|align="center"|
|align="center"|90%
|align="center"|
|align="center"|0.9
|align="left"|
|align="left"|
Implementation of Amplitude
Implementation of Amplitude
|
|
*Went through Amplitude onboarding.
*Should be collecting real-time data this week.
*Have gotten the ball rolling with coordination.
*We’ve haven’t completely tied everything off, but it’s basically done.
*Working in stage and currently fine tuning.
|-
|-
|}
|}

Latest revision as of 01:53, 30 September 2017

September 25, 2017

Q3 OKR Progress

Objective Key Result Confidence Features Notes
Jul 10 Jul 24 Aug 8 Aug 21 Sep 5 Sep 18 Score
1. To have improved the quality for the Firefox Account experience on desktop. 1.1 Close ALL (100%) issues in the Quantum/Photon milestone. (Crouton) 60% 65% 70% 75% 90% 100% 1.0

Priorities for support of Quantum/Photon in 57:

1.2 Close 50% of our Q3 quality backlog. 50% 75% 75% 90% 100% 1.0

Quality Priorities for Q3:

1.3 Increase by 15% the rate of users that become multi-device as a result of going through our first run flow. 50% 50% 50% 40% 0% 0% 0.0
  • Didn't achieve this. Had a lot of questions about the metrics.
1.4 Fix newsletter subscription when email is confirmed on a 2nd device. 95% 95% 95% 95% 90% 90% 1.0

Fixes a problem with getting marketing newsletter opt-in data over to Salesforce.

  • Done!
2. To have improved the quality of the Firefox Account experience on mobile. 2.1 Measure and A/B test best push opt-in screen on iOS. 50% 40% 55% 55% 50% 50% 0.7
  • PR reviewed but not merged.
  • Review brought up some interesting points to address.
2.2 Complete 80% of Q3 mobile bugs. N/A 60% 60% 60% 65% 90% 0.8

Identification and fixing of the most impactful bugs on iOS

3. To have completed all features that are still in flight from Q1 and Q2. 3.1 Have a device pairing flow ready to A/B test. 75% 75% 50% 25% 0% 0% 0.3
  • Allow users to test out device pairing. This does not necessarily mean that QR codes are used.
  • Pushed forward the challenge with auto-focus.
  • There are things we got done this quarter that we can roll forward.
  • Great analysis.
3.2 Ship secondary email. 80% 80% 90% 90% 90% 90% 0.9
  • We are allowing users to add a secondary email. Allows users to change primary email.
  • Caching bug fixed
  • Client code is in
  • Works for AMO, and they have an issue to fix one little thing
  • Pocket was the only remaining thing.
3.3 Land support for relier encryption keys. 70% 70% 75% 75% 75% 80% 0.9

Relier encryption keys

3.4 Integrate 100% of high priority Amplitude events. 75% 75% 75% 75% 75% 90% 0.9

Implementation of Amplitude

  • Should be collecting real-time data this week.
  • We’ve haven’t completely tied everything off, but it’s basically done.
  • Working in stage and currently fine tuning.