Identity/Metrics

From MozillaWiki
Jump to navigation Jump to search
Last updated: 2014/02/21

Firefox Accounts

Stakeholders

  • Product Managers, including PMs for services that rely on FxA: what does FxA adoption look like?
  • UX: are users successful, where do they get stuck?

Nutshell Summary

Quick summary of what's planned (in priority order).

Infrastructure

  • fxa-auth-server (and fxa-content-server) log events locally
  • heka processes those logs and sends data to log aggregator
  • more heka processing at the log aggregator, which creates elasticsearch indexes and mappings in elasticsearch db (as well as anonymization and other aggregation)
  • kibana dashboard(s) access elasticsearch db (all metrics described below can be expressed as counts/time)
  • log aggregator, elastic search & kibana are either on shared services, or an FxA specific mirror of that setup
  • a development environment will exist that has mirror of production data (to hack on heka filters, elasticsearch mappings, kibana dashboards etc.)

0. One Metric That Matters: Total number of Firefox Accounts

  • Web page that Mark can show to anyone -- this is the top line goal for FxA
  • At minimum, could be a single number
  • Display a graph over time

1. Accounts created per day

  • Also logins per day, and other events that correspond to fxa-auth-server endpoints
  • segmented by device (Android, Desktop, FxOS, browser), service (Marketplace, Sync, WMF, Settings), locale
  • derived from fxa-auth-server logs: log line at each endpoint contains the raw information, processed by heka, data eventually lands in an elasticsearch index.
  • device is derived from user-agent
  • locale is sent on request header
  • service is passed in on some endpoints

2. Active daily users

  • derived from fxa-auth-server logs: # calls to cert/sign endpoint, bucketed by userid.
  • alternative could be derived from verifier logs, again bucketed by userid
  • heka can do processing to count unique userid's/day, so that userid is not stored in elastic search
    • note: The sandbox is intentionally resource constrained, it is highly unlikely it will have enough memory to track every active user in a day as usage increases. -trink.

3. UX flow analysis

  • count number of people who make it to each stage of a given UX flow (per day)
  • note that it is not important to connect events in a flow for an individual user: associate each event with a flow type (create account, login, reset password) and assume that the flow happens in the context of one day
  • derived from fxa-auth-server logs initially, can add in events from fxa-content-server logs, etc. eventually (as long as the "flow type" matches)
  • not 100% clear to me yet whether we can identify the "flow" in the server, or if we need help by information passed in from the client
  • segmentations come along for the ride: service, device, locale

Additional segmentations

  • UX would like screen size segmentations, would require info from the client/device

UX Flow Timing Information

  • It would be great to get timing information for each stage in a UX flow (ms since the "start" of the flow), but this requires help from the client.
  • Getting great visualizations from this data might be a challenge in kibana (without some custom hacking)

Additional stages for UX flows, additional UX flows

  • Would be great to track # people who have opportunity to create account but then skip it (in FTU flows)
  • Would be great to tie email validation stage into FTU flow analysis (happens at different point in time -- possibly even on a different day)
  • Create and login flows are prioritized. Once things are up and running it should be easy to add password reset, etc.

Sync specific metrics

User Stories

Dependencies

Timeline

Resources

Older Notes

Sync.next

Persona

Retrieved from "https://wiki.allizom.org/index.php?title=Identity/Metrics&oldid=934641"