Latest revision as of 17:55, 15 December 2015

A Public ElasticSearch Cluster with Bugzilla Data

Objective

Provide the public with an easy-to-query historical view of Bugzilla bugs.

Justification

The BMO/ES cluster is important for managing many of Mozilla's projects: It is an active data system that underlies a multitude of dashboards to track various project metrics. It is capable of providing both high level aggregates over the million+ issues, providing detailed historical views into those issues, and it provides that information fast.

Let's Use It!

BMO/ES is a service, which means you can use it right now! You probably already have your browser open, so let's send it queries:

Examples

Minimum required to hit ES cluster on Github

Minimum required to pull comments on Github

Simple Bug Count on Github

Can you reach the private cluster? on Github

Basic walk-through

https://github.com/klahnakoski/Qb/blob/master/docs/BZ_Tutorial.md

Code

Backend ETL Code: https://github.com/klahnakoski/Bugzilla-ETL

Frontend Examples: https://github.com/klahnakoski/MoDevMetrics/tree/dev

Hacking/Contributors

This project needs your help! If you love data munging and dealing with complexity; and you get satisfaction that people will build atop your hard work every day, then this project is for you!

Contacts

IRC: ekyle@irc.mozilla.org (in #ateam and #bteam)
email: klahnakoski@mozilla.com

Good First Bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Big bugs

Full Query

ID	Summary	Status
997228	Renaming flags will destroy bug history for those flag values	NEW
1125904	Blocked/blocked_by/depends_on is too confusing	NEW
1197813	[BZ ETL] Tracking flags changed corrupt bug history	NEW
1201064	cf_blocking_loop dropped (renamed), not reflected in history	NEW

4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);

@@ Line 1: / Line 1: @@
-= Publicize ElasticSearch Cluster with Bugzilla Data =
+= A Public ElasticSearch Cluster with Bugzilla Data =
 == Objective ==
-Have a publicly accessible ElasticSearch cluster containing all historical data on non-sensitive bugs.  The hope is this data will attract community and uncover interesting trends.
+Provide the public with an easy-to-query historical view of Bugzilla bugs.
-This project is a small element of a [http://mrcote.info/blog/2014/01/03/bmo-in-2013/ greater battery of improvements planned for 2014]
+== Justification ==
-== What's it Good For? ==
+The BMO/ES cluster is important for managing many of Mozilla's projects:  It is an [https://wiki.mozilla.org/Projects/ActiveData active data] system that underlies a multitude of dashboards to track various project metrics.  It is capable of providing both high level aggregates over the million+ issues, providing detailed historical views into those issues, and it provides that information ''fast''.
-A [http://people.mozilla.org/~klahnakoski/docs/Bugs%20in%20ES.pdf ppt presentation] to show what you can build using this cluster as a [http://en.wikipedia.org/wiki/Data_mart datamart].
+= Let's Use It! =
-== Let's Use It! ==
+BMO/ES is a service, which means you can use it right now!  You probably already have your browser open, so let's send it queries:
-'''Code'''
-:Backend ETL Code: https://github.com/klahnakoski/Bugzilla-ETL
+==Examples==
-:Frontend Examples: https://github.com/klahnakoski/MoDevMetrics/tree/dev
-'''Examples'''
 :[http://people.mozilla.org/~klahnakoski/modevmetrics/Tutorial01-Minimum.html Minimum required to hit ES cluster] [https://github.com/klahnakoski/MoDevMetrics/blob/dev/html/Tutorial01-Minimum.html on Github]
 :[http://people.mozilla.org/~klahnakoski/modevmetrics/Tutorial02-Comments.html Minimum required to pull comments] [https://github.com/klahnakoski/MoDevMetrics/blob/dev/html/Tutorial02-Comments.html on Github]
 :[http://people.mozilla.org/~klahnakoski/modevmetrics/Tutorial03-Bug-Count.html Simple Bug Count] [https://github.com/klahnakoski/MoDevMetrics/blob/dev/html/Tutorial03-Bug-Count.html on Github]
-'''Basic walk-through'''
+:[http://people.mozilla.org/~klahnakoski/modevmetrics/Tutorial04-Private.html Can you reach the private cluster?] [https://github.com/klahnakoski/MoDevMetrics/blob/dev/html/Tutorial04-Private.html on Github]
+==Basic walk-through==
 :https://github.com/klahnakoski/Qb/blob/master/docs/BZ_Tutorial.md
+==Code==
+:Backend ETL Code: https://github.com/klahnakoski/Bugzilla-ETL
+:Frontend Examples: https://github.com/klahnakoski/MoDevMetrics/tree/dev
+= Hacking/Contributors =
+This project needs your help!  If you love data munging and dealing with complexity; and you get satisfaction that people will build atop your hard work every day, then this project is for you!
+== Contacts ==
+* '''IRC:''' ekyle@irc.mozilla.org (in #ateam and #bteam)
+* '''email:''' klahnakoski@mozilla.com
+== Good First Bugs ==
+<div>
+<bugzilla>
+{
+"include_fields": "id, summary, status, resolution, assigned_to",
+"resolution":"---",
+"f1":"blocked",
+"o1":"equals",
+"v1":959670,
+"f2":"status_whiteboard",
+"o2":"anywordssubstr",
+"v2":"[good first bug]"
+}
+</bugzilla>
+</div>
+== Big bugs ==
+<div>
+<bugzilla>
+{
+"include_fields": "id, summary, status, resolution, assigned_to",
+"resolution":"---",
+"f1":"blocked",
+"o1":"equals",
+"v1":959670,
+"f2":"status_whiteboard",
+"o2":"nowordssubstr",
+"v2":"[meta]"
+}
+</bugzilla>
+</div>
 == Other Reading ==
-* History: https://wiki.mozilla.org/BMO/ElasticSearch/History
+* [https://wiki.mozilla.org/BMO/ElasticSearch/History Adventures in initial BMO/ES development]
+* [https://wiki.mozilla.org/BMO/ElasticSearch/History#New_Design Architecture] included in part of the adventures
+* A [http://people.mozilla.org/~klahnakoski/docs/Bugs%20in%20ES.pdf ppt presentation] to show what you can build using this cluster as a [http://en.wikipedia.org/wiki/Data_mart datamart].
+[[Category:BMO]]

BMO/ElasticSearch: Difference between revisions