Confirmed users
9,511
edits
EngineeringProductivity/Projects/ActiveData: Difference between revisions
EngineeringProductivity/Projects/ActiveData (view source)
Revision as of 05:34, 19 September 2019
, 19 September 2019→Overview
Klahnakoski (talk | contribs) m (→Dependencies: typo) |
|||
| (One intermediate revision by one other user not shown) | |||
| Line 4: | Line 4: | ||
ActiveData is a collection of about 8 billion records (Feb 2016) covering unit tests, Buildbot jobs, performance data, and mercurial. This collection is publicly available, and can be queried directly, similar to any database. | ActiveData is a collection of about 8 billion records (Feb 2016) covering unit tests, Buildbot jobs, performance data, and mercurial. This collection is publicly available, and can be queried directly, similar to any database. | ||
ActiveData is built on top of ElasticSearch, a fast, distributed, redundant document store. ActiveData provides the benefits of familiar and succinct SQL by translating SQL-like queries to ElasticSearch queries | ActiveData is built on top of ElasticSearch, a fast, distributed, redundant document store. ActiveData provides the benefits of familiar and succinct SQL by translating SQL-like queries to ElasticSearch queries. | ||
== Problem == | == Problem == | ||
| Line 11: | Line 11: | ||
== Solution== | == Solution== | ||
ActiveData will serve as a reusable ETL pipeline; annotating the test results with as much relevant data as possible. It also provides a query service to explore and aggregate the data, so there is minimal setup required to access this data. | ActiveData will serve as a reusable ETL pipeline; annotating the test results with as much relevant data as possible. It also provides a query service to explore and aggregate the data, so there is minimal setup required to access this data. | ||
= Redash = | = Redash = | ||
| Line 115: | Line 114: | ||
ActiveData's primary goal is to support dashboards that give Mozilla useful perspectives into the large amount of data: | ActiveData's primary goal is to support dashboards that give Mozilla useful perspectives into the large amount of data: | ||
* [https://github.com/mozilla/active-data-recipes/tree/master/adr/recipes ActiveData Recipes has a variety of use cases] | |||
* Individual unit test results | * Individual unit test results | ||
* | * Task cluster test timing | ||
* Firefox compile times | * Firefox compile times | ||
* Recently new, removed, and disabled tests | * Recently new, removed, and disabled tests | ||
* Buildbot wait times | * Buildbot wait times | ||
* CodeCoverage aggregates and per-file detail | |||
= Let's Use It! = | = Let's Use It! = | ||