CloudServices/DataPipeline

Overview

The cloud services data pipeline ingests data for analysis, monitoring and reporting. The pipeline is currently used for processing desktop and device Telemetry data and cloud services server logs. The ingestion pipeline is one component of the Fx Data Platform.

Pipeline specs/docs

• Pipeline technical proposal
• HTTP Edge Server Specification
• Pipeline Metadata

Data sets and other documentation

• Telemetry Data
• Mobile Metrics
• Test Pilot

Code

V2 Pipeline

Link	Description
https://github.com/mozilla-services/data-pipeline	Mozilla Services Data Pipeline
https://github.com/mozilla-services/lua_sandbox	Generic Lua sandbox for dynamic data analysis
https://github.com/mozilla-services/mozilla-pipeline-schemas	JSON Schema specifications of pipeline data
https://github.com/mozilla/pipeline-monitoring-dashboard	Monitoring data quality issues for metrics pipeline
https://github.com/mozilla-services/heka	Data collection and processing made easy
https://github.com/mozilla-services/nginx_moz_ingest	HTTP Data Pipeline Ingestion
https://github.com/trink/hindsight	Data collection and processing made light weight, fast, and more reliable

Telemetry

Link	Description
https://github.com/vitillo/telemetry-onboarding	Slides / notebooks for Telemetry Onboarding
https://github.com/mozilla/telemetry-server	Code for analysis.telemetry.mozilla.org among other things
https://github.com/bsmedberg/telemetry-experiments-dashboard	A dashboard to track the deployment of Firefox Telemetry Experiments
https://github.com/mozilla/telemetry-batch-view	A Scala framework to build derived datasets, aka batch views, of Telemetry data.
https://github.com/mozilla/cerberus	Automatic alert system for telemetry histograms
https://github.com/mozilla/emr-bootstrap-spark	AWS bootstrap scripts for Mozilla's flavoured Spark setup.
https://github.com/mozilla/moz-crash-rate-aggregates	Crash Rate Aggregation code
https://github.com/mozilla/jupyter-notebook-gist	Plugin to create, list, and load GitHub Gists from Jupyter notebooks
https://github.com/mozilla/jupyter-spark	Jupyter Notebook extension for Apache Spark integration
https://github.com/mozilla/python_mozaggregator	Aggregator job for telemetry.mozilla.org
https://github.com/mozilla/python_moztelemetry	Spark bindings for Mozilla Telemetry
https://github.com/mozilla/telemetry-analysis-service	Eventual home of the revamped a.t.m.o (per Bug 1248688)
https://github.com/vitillo/telemetry-airflow	Scheduling / workflow management for Telemetry jobs
https://github.com/vitillo/e10s_analyses	Data analysis relating to Electrolysis / E10s
https://github.com/mozilla/telemetry-tools	Utility code to work with Mozilla Telemetry data

Archive

• Q4 2014: Reporting and monitoring overview
• Bespoke Dashboards
• Cloud Services Data Projects
• List of Data Sources
• V1 Pipeline & Data Sources
• post workweek roadmap
• old etherpad