Loop/Data Collection

To monitor the performance and other aspects of the Loop service, we will be gathering information independent of the platform metrics. The list of collected metrics will be expanded in later phases of the project.

Mozilla has a number of data collection services, including:

• Telemetry
• Log Collection
• Crash Reporting
• Input

The collected data is broken down into these categories below.

Telemetry Metrics

For simple telemetry that can be indicated by a simple scalar or boolean, we will be using the nsITelemetry interface.

For custom collected telemetry, the basic steps to be performed are:

1. Create a report object as detailed below, setting the "report" field to indicate which kind of report is being submitted
2. Create a new UUID-based report ID ("slug")
3. Save the created report to a file in {PROFILE_DIRECTORY}/saved-telemetry-pings, in a file named after the report ID.

In general, this report will look something like the following:

{
  "reason": "loop",
  "slug": "75342750-c96c-11e3-9c1a-0800200c9a66",
  "payload": {
    "ver": 1,
    "info": {
      "appUpdateChannel": "default",
      "appBuildID": "20140421104955",
      "appName": "Firefox",
      "appVersion": "32.0",
      "reason": "loop",
      "OS":"Darwin",
      "version":"12.5.0"
    },
    "report": "...",
    // Fields unique to the report type are included here.
  }
}

ICE Failures

Tracked in bug 998989

One of the first and most important bits of information we need to gather is ICE failure information. When we initially launch the MLP, it is quite likely that we will see non-trivial connection failures, and we will want to be in a position to rapidly diagnose (and, where possible, fix) the causes of these failures. To that end, we need to be able to get to the ICE statistics; and, if a call fails to set up (i.e., ICE connection state transitions to "failed" or "disconnected"), upload ICE logging information.

Details

1. When a PeerConnection is instantiated, begin listening for the iceconnectionstatechange event
   • This may require adding code in PeerConnection.js that allows retrieval of all of the PCs for a given WindowID, using the _globalPCList
2. If the state changes to "failed" or "disconnected", Retrieve ICE logging information and upload it to the ICE log to the telemetry server:
   1. Instantate a WebrtcGlobalInformation object (see http://dxr.mozilla.org/mozilla-central/source/dom/webidl/WebrtcGlobalInformation.webidl)
   2. Call "getLogging" on the WGI object
   3. In the callback from getLogging (which takes an array of log lines as its argument):
      1. Consolidate the log file into a single string
      2. Submit the report as detailed above

Example payload

{
    "ver": 1,
    "info": {
      ... // See example at top of page
    },
    "report": "ice failure",
    "connectionstate": "failed",
    "stats": <...JSONified version of RTCStatsReport...>
    "localSdp": <...SDP body...>,
    "remoteSdp": <...SDP body...>,
    "log": <...~50-1000 kB of log file...>
}

(See also the "payload" field of https://bug998989.bugzilla.mozilla.org/attachment.cgi?id=8446879)

Nature of Data

• Uncompressed, these logs are on the order of 50-1000 kB, but they contain significant redundancy: when run through gzip, they reduce to something more in the 3-60 kB range.
• We anticipate that these logs will be generated for something on the order of 5% to 20% of total call volume initially, shrinking as we troubleshoot and eliminate the issues they help us discover.
• When the feature lands in nightly, we are optimistically hoping for on the order of 10,000 users, probably making no more than 1 or 2 calls a day (i.e., we would anticipate initial worst-case load to be approximately 4,000 logs per day). After we get some experience with the feature, we should be able to refine this to be more accurate.
• The logs will necessarily contain users' IP addresses (and will be nearly useless without them) IP address information will be fuzzed; but information in the SDP extends the fingerprinting surface, so access control will be important.
• Eventually, we will want to develop fingerprinting heuristics for grouping logs together that are likely to share common root causes, but this is not required for the initial deployment.
• For initial analysis, we could probably do with something as simple as a report that says "on date, there were x failures, broken down as follows: failed: failed count, disconnected: disconnected count," and then lets us list all the failures for a given date/reason pair, ultimately allowing us to download the log to analyze. As we get experience with how things tend to break, we might want to refine this some, but it's a good start.

Call Quality Metrics

This covers collection of the following information:

• Call setup latency
• Media metrics (RTT, jitter, loss)

Tracked in bug 1003163

URL Sharing

Tracked in bug 1015988

Input Metrics

The API for submitting data to input.mozilla.org is described here.

User Satisfaction

Tracked in bug 1003180

Based on bug 974873, the current feedback proposal is as follows:

• Screen with Happy or Sad displayed at the end of the call in the conversation window
  • If happy clicked, thank the user for the feedback and close conversation window
  • If Sad clicked display another screen:
    • Audio Quality (category: "audio")
    • Video Quality (category: "video")
    • Was Disconnected (category: "disconnected")
    • Confusing (category: "confusing")
    • Other (w/ text field) (category: "other")

Example payload

{
    "happy": true,
    "description": "",
    "product": "Loop",
    "url": "http://call.mozilla.org/", // designates this as a link-clicker
}

{
    "happy": false,
    "category": "other",
    "description": "Could not move video window out of the way",
    "product": "Loop"
}

This format is changing

{
    "ver": 1,
    "info": {
      ... // See example at top of page
    },
    "report": "user satisfaction",
    "client": "standalone",
    "feedback": "happy"
}

{
    "ver": 1,
    "info": {
      ... // See example at top of page
    },
    "report": "user satisfaction",
    "feedback": "sad",
    "client": "builtin",
    "reason": "other",
    "textField": "Could not move video window out of the way"
}

Other Metrics

The following data types don't clearly match existing data collection systems. They may be able to leverage existing systems, possibly with modifications; or they may require entirely new data ingestion servers.

User Issue Reports

Tracked in bug 1024568

This needs further fleshing out; it is currently a placeholder

We should ideally have a mid-call mechanism to say "this call has an issue," which will submit a report of the following form:

• UA string of remote party (TBD: how to collect?)
• Console log from duration of call
• All the information from the ice failure reports, above
• /tmp/WebRTC.log, after performing the same operations as the "start debug mode" / "stop debug mode" buttons do