Baloo

Intro

Project Baloo, is a collaborative effort between the Data and BI Services and Community Building team to create a contribution tracking system for Mozilla.

Roadmap

File a Baloo bug

Integration Status

System	Area	Priority	Schema Update	Conversion points	Import of data	Building Aggregations	On 10x dashboard
Support.mozilla.org	SUMO	P1	Yes	980385	1010197	Done	Done
bugzilla.mozilla.org	QA	P1	990667	1019689	1010205	Done	Done
Bugzilla and Github	Coding	P1	1010187	1010190	1010209	Done	1025952
reps.mozilla.org	Mozilla Reps	P1	1010180	Done	1010183	1010207	Done
developer.mozilla.org	MDN	P1	see Q2 above	1050315	see Q2 above	see Q2 above	see Q2 above
addons.mozilla.org	AMMO	P1		1050317
marketplace.mozilla.org	AMMO	P1		1050317
github.com	WebDev	P2	Done	Done	Done (MoFo DB)	1056771	Done
location.services.mozilla.com	Location Services	P1		1073054
webcompat.com	Web Compatibility	P2		1050320
moztrap.mozilla.org	QA	P2	1042101	Done	1079804	1079803
oneanddone.mozilla.org	QA	P1	1049696	1050321	1069250	1069250
mofointegration DB	Webmaker	P1	Schema	Done	1047299	1047309
mofointegration DB	Science Lab	P2	Schema	Done	1047299	1047313
mofointegration DB	Open News	P2	Schema	Done	mofo bug	mofo bug
mofointegration DB	MoCo Ad-hoc	P1	Schema	1047299	1047319
mofointegration DB	MozFest	P2	Schema	Done	1047299	1047315

Done

In process

Blocked

No action yet

Q1 2014

• Establish Baloo documentation and wiki pages -- DONE
• Creation of appropriate bagheera entry points -- DONE
• Tryout backporting of historical data from a major contribution system (SuMo?) -- DONE

Q2 2014

• Start production backporting of historical data from all major contribution systems -- DONE
• Creation of first dashboard based on 2013 historical data -- DONE

Q3 2014

Full Query

ID	Summary	Priority	Status
1042101	[Baloo] [QA] Schema update to include Moztrap	--	RESOLVED
1047299	Import data from mofointegration to Baloo	--	RESOLVED
1047309	Build Baloo aggregations for Webmaker	--	RESOLVED
1047313	Build Baloo aggregations for Science Lab	--	RESOLVED
1047315	Build Baloo aggregations for MozFest	--	RESOLVED
1047319	Build Baloo aggregations for MoCo teams using Ad-hoc logger	--	RESOLVED
1049696	[Baloo] [QA] Schema update to include One and Done	--	RESOLVED
1050315	[Baloo] Create MDN conversion points	--	RESOLVED
1050317	[Baloo] Create AMMO conversion points	--	NEW
1050320	[Baloo] Create Webcompat conversion points	--	NEW
1050321	[Baloo] Create One and Done conversion points	--	RESOLVED
1053149	Simplify transfer of Baloo data from Tableau to Dashboard	--	RESOLVED
1053643	[Baloo] Auto updating of Baloo data exports	--	NEW
1056771	[Baloo] Automating github aggregations from MoFo integration DB	--	RESOLVED
1056909	[Baloo] Update Schema to include github issues	--	RESOLVED
1059885	Create a requirements document for areweamillionyet.org	--	RESOLVED
1069250	[Baloo] Build and automate aggregations for One and Done	--	RESOLVED
1073054	[Baloo] Create Location Services conversion points	--	RESOLVED

18 Total; 3 Open (16.67%); 15 Resolved (83.33%); 0 Verified (0%);

Q4 2014

Full Query

ID	Summary	Priority	Status
1047299	Import data from mofointegration to Baloo	--	RESOLVED
1047309	Build Baloo aggregations for Webmaker	--	RESOLVED
1047313	Build Baloo aggregations for Science Lab	--	RESOLVED
1047315	Build Baloo aggregations for MozFest	--	RESOLVED
1047319	Build Baloo aggregations for MoCo teams using Ad-hoc logger	--	RESOLVED
1050315	[Baloo] Create MDN conversion points	--	RESOLVED
1050317	[Baloo] Create AMMO conversion points	--	NEW
1050320	[Baloo] Create Webcompat conversion points	--	NEW
1053643	[Baloo] Auto updating of Baloo data exports	--	NEW
1059885	Create a requirements document for areweamillionyet.org	--	RESOLVED
1069250	[Baloo] Build and automate aggregations for One and Done	--	RESOLVED
1073054	[Baloo] Create Location Services conversion points	--	RESOLVED
1079803	[Baloo] Build and automate aggregations for Moztrap	--	RESOLVED
1079804	[Baloo] Import data from moztrap to Baloo	--	RESOLVED

14 Total; 3 Open (21.43%); 11 Resolved (78.57%); 0 Verified (0%);

Technical Documentation

Baloo is re-using Data and BI Services team technological stack re-purposed to suit the Contributor tracking needs. Have a look on the data flow diagram below to understand the inner workings.

Stack

Schema

Baloo deals with Contribution Data. Contribution Data are defined using the established Schema of the Community Building Team.

Adding a new system

Tracking contributions without adding a new system
Some contribution areas don't have contribution systems or their systems are not eligible for Baloo integration because they track a low volume of contributors (such as Creative, Legal and Policy). Check Ad-Hoc Baloo solution for an alternative way to track contributions in those areas

Several steps are needed in order to add a new system in Baloo. They can be broken down to two different sections. Porting historical data, and implementing live pushes

Adding Historical Data

Once a system is identified holding historical contribution data, the following should be done to import those to Baloo. Note that this should be an one off procedure given that live-pushes are also implemented in a system.

1. Identify *what* do you want to track in your system. Write down all possible contributions that are happening in your area.
2. Talk to your principal webdev or technical people, on how this info is stored in your database.
3. Tweak/amend the Contribution Schema to fit your tracking needs. (Contact Pierros or join Systems and Data WG meeting for help)
4. Open a new bug on Mozilla Metrics::Data/Backend Reports describing the location of the data and the schema/mapping your are about to use
5. Data and BI team will be following up with you on imports

Adding Live Pushes

1. Make sure you follow steps 1,2 and 3 above.
2. Coordinate with your principal webdev or technical people on implementing the code on your system side that produces and pushes the JSON contribution tracking payloads.
3. Open a new bug on Mozilla Metrics::Data/Backend Reports so we can track the implementation process.
4. Tryout staging the code using the following

   curl -v -X POST -d '{"test":"hello"}' http://10.22.28.101:8080/submit/baloo/

   If the operation successful we will return 201.
5. Once you are ready for production you may use the production url: https://data.mozilla.com/submit/baloo/

All data is in raw format till this point
If any or both of the above is implemented for your system, you have raw data in our HBASE. Although this is available for crunching, you still need to establish your contribution area conversion points so we can start tracking active contributors

Tracking active contributors

Once one or both of the processes above are complete we now have raw contribution data about your area. Raw data is cool, but we need to start making sense out of them. For that reason we need to establish Conversion Points to understand the type of contributions and identify Active contributors.

1. Have a meeting with your CBT contact person to identify (if not already done) your contribution pathway.
2. Navigate to our Conversion Points wiki page and add/edit your contribution area with all relevant info.
3. Once the conversion points are up to date, open a bug on Mozilla Metrics::Data/Backend Reports so we can create the aggregation scenarios and start crunching the raw data.
4. Magical numbers will flow in, and become available in Tableau (and APIs!) for usage!

Meetings

• Meeting 2014-10-30
• Meeting 2014-10-16
• Meeting 2014-10-09
• Meeting 2014-10-02
• Meeting 2014-09-25
• Meeting 2014-09-18
• Meeting 2014-09-11
• Meeting 2014-09-04
• Meeting 2014-08-21
• Meeting 2014-08-14
• Meeting 2014-08-07
• Meeting 2014-07-31
• Meeting 2014-07-10
• Meeting 2014-07-03
• Meeting 2014-06-16

You can also see meeting notes on the master etherpad

Contact

Pierros and Adam serve as the main contact people at this point. Reach out to them for further info and instructions on how to add your system to Baloo.

You may also find us on #baloo IRC channel in irc.mozilla.org