Loop/Data Collection

To monitor the performance and other aspects of the Loop service, we will be gathering information independent of the platform metrics. The list of collected metrics will be expanded in later phases of the project.

Mozilla has a number of data collection services, including:

• Telemetry
• Log Collection
• Crash Reporting
• Input

The collected data is broken down into these categories below.

Telemetry Metrics

For simple telemetry that can be indicated by a simple scalar or boolean, we will be using the nsITelemetry interface.

For custom collected telemetry, the basic steps to be performed are:

1. Create a report object as detailed below, setting the "report" field to indicate which kind of report is being submitted
2. Create a new UUID-based report ID ("slug")
3. Save the created report to a file in {PROFILE_DIRECTORY}/saved-telemetry-pings, in a file named after the report ID.

In general, this report will look something like the following:

{
  "reason": "loop",
  "slug": "75342750-c96c-11e3-9c1a-0800200c9a66",
  "payload": {
    "ver": 1,
    "info": {
      "appUpdateChannel": "default",
      "appBuildID": "20140421104955",
      "appName": "Firefox",
      "appVersion": "32.0",
      "reason": "loop",
      "OS":"Darwin",
      "version":"12.5.0"
    },
    "report": "...",
    // Fields unique to the report type are included here.
  }
}

ICE Failures

Tracked in bug 998989

One of the first and most important bits of information we need to gather is ICE failure information. When we initially launch the MLP, it is quite likely that we will see non-trivial connection failures, and we will want to be in a position to rapidly diagnose (and, where possible, fix) the causes of these failures. To that end, we need to be able to get to the ICE statistics; and, if a call fails to set up (i.e., ICE connection state transitions to "failed" or "disconnected"), upload ICE logging information.

Details

1. When a PeerConnection is instantiated, begin listening for the iceconnectionstatechange event
   • This may require adding code in PeerConnection.js that allows retrieval of all of the PCs for a given WindowID, using the _globalPCList
2. If the state changes to "failed" or "disconnected", Retrieve ICE logging information and upload it to the ICE log to the telemetry server:
   1. Instantate a WebrtcGlobalInformation object (see http://dxr.mozilla.org/mozilla-central/source/dom/webidl/WebrtcGlobalInformation.webidl)
   2. Call "getLogging" on the WGI object
   3. In the callback from getLogging (which takes an array of log lines as its argument):
      1. Consolidate the log file into a single string
      2. Submit the report as detailed above

Example payload

See https://bug998989.bugzilla.mozilla.org/attachment.cgi?id=8446879

Nature of Data

• Uncompressed, these logs are on the order of 50-1000 kB, but they contain significant redundancy: when run through gzip, they reduce to something more in the 3-60 kB range.
• We anticipate that these logs will be generated for something on the order of 5% to 20% of total call volume initially, shrinking as we troubleshoot and eliminate the issues they help us discover.
• When the feature lands in nightly, we are optimistically hoping for on the order of 10,000 users, probably making no more than 1 or 2 calls a day (i.e., we would anticipate initial worst-case load to be approximately 4,000 logs per day). After we get some experience with the feature, we should be able to refine this to be more accurate.
• The logs will necessarily contain users' IP addresses (and will be nearly useless without them) IP address information will be fuzzed; but information in the SDP extends the fingerprinting surface, so access control will be important.
• Eventually, we will want to develop fingerprinting heuristics for grouping logs together that are likely to share common root causes, but this is not required for the initial deployment.
• For initial analysis, we could probably do with something as simple as a report that says "on date, there were x failures, broken down as follows: failed: failed count, disconnected: disconnected count," and then lets us list all the failures for a given date/reason pair, ultimately allowing us to download the log to analyze. As we get experience with how things tend to break, we might want to refine this some, but it's a good start.

Call Quality Metrics

This covers collection of the following information:

• Call setup latency
• Media metrics (RTT, jitter, loss)

Tracked in bug 1003163

URL Sharing

Tracked in bug 1015988

Input Metrics

The API for submitting data to input.mozilla.org is described here. We will be using a "product" value of "Hello" (cf. Bug 1003180, comment 29).

User Satisfaction

Tracked in bug 1003180 (meta) and:

• bug 972992 (built-in client)
• bug 974873 (standalone client).

The built-in client will have a screen similar to this pop up when the user finishes a call:

The link-clicker client will have a similar screen for call completion.

Based on bug 974873, the current feedback proposal is as follows:

• Screen with Happy or Sad displayed at the end of the call in the conversation window
  • If happy clicked, thank the user for the feedback and close conversation window
  • If Sad clicked display another screen:
    • Audio Quality (category: "audio")
    • Video Quality (category: "video")
    • Was Disconnected (category: "disconnected")
    • Confusing (category: "confusing")
    • Other (w/ text field) (category: "other")

For the standalone client, we will report the "happy" boolean; and, if false, indicate the reason using the "category" field and the user-provided input in the "description" field. We will also include the URL that the user clicked on to initiate the call (without the call token) in the "url" field. The key reason for doing so is that it allows us to distinguish standalone feedback from build-in client feedback. Secondarily, if we change URL endpoints (say, due to versioning), it allows us to distinguish reports from two different kinds of server. Because of the shortcomings of the individual fields in the standard userAgent object, we will report the entire user agent string for the standalone client (i.e., the value in navigator.userAgent).

For the built-in client, we will report the "happy" boolean; and, if false, indicate the reason using the "category" field and the user-provided input in the "description" field. We will also report channel, version, and platform; see the fjord API documentation for more information about these fields.

Example Payloads

An example payload for a standalone ("link-clicker") client:

POST https://input.allizom.org/api/v1/feedback HTTP/1.1
Accept: application/json
Content-type: application/json

{
    "happy": true,
    "description": "",
    "product": "Loop",
    "url": "http://call.mozilla.org/", // designates this as a link-clicker
    "user_agent": "Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0"
}

An example payload for a built-in client:

POST https://input.allizom.org/api/v1/feedback HTTP/1.1
Accept: application/json
Content-type: application/json

{
    "happy": false,
    "category": "other",
    "description": "Could not move video window out of the way",
    "product": "Loop",
    "channel": "aurora",
    "version": "33.0a1",
    "platform": "Windows 8"
}

Other Metrics

The following data types don't clearly match existing data collection systems. They may be able to leverage existing systems, possibly with modifications; or they may require entirely new data ingestion servers.

User Issue Reports

Tracked in bug 1024568

This needs further fleshing out; it is currently a placeholder

We should ideally have a mid-call mechanism to say "this call has an issue," which will submit a report of the following form:

• UA string of remote party (TBD: how to collect?)
• Console log from duration of call
• All the information from the ice failure reports, above
• /tmp/WebRTC.log, after performing the same operations as the "start debug mode" / "stop debug mode" buttons do