Baloo

Intro

Project Baloo, is a collaborative effort between the Data and BI Services and Community Building team to create a contribution tracking system for Mozilla.

Status

<smartsheet id="bdc633a5864c43ff9c07893c947e2a29" width="100%" height="400px" />

Roadmap

Q1 2014

• Establish Baloo documentation and wiki pages -- DONE
• Creation of appropriate bagheera entry points -- DONE
• Tryout backporting of historical data from a major contribution system (SuMo?) -- DONE

Q2 2014

• Start production backporting of historical data from all major contribution systems -- ONGOING

System	Area	Providing data	Integrating with Baloo	Calculating contributors	On 10x dashboard
Support.mozilla.org	SUMO	Yes	Yes	In process
Bugzilla.mozilla.org	QA		bug 990666

• • developer.mozilla.org
  • Reps.mozilla.org
  • mozilla-central
  • Addons.mozilla.og
• Creation of first dashboard based on 2013 historical data

Q3 2014

• Deploy live pushes on all major systems
• Deploy live pushes on all Mozilla systems where contributions are happening
• Creation of custom contribution tracking interface for contribution areas without established systems

Q4 2014

• 100% coverage of contribution systems across mozilla (including the manual non-automated ones)

Technical Documentation

Baloo is re-using Data and BI Services team technological stack re-purposed to suit the Contributor tracking needs. Have a look on the data flow diagram below to understand the inner workings.

Stack

Schema

Baloo deals with Contribution Data. Contribution Data are defined using the established Schema of the Community Building Team.

Adding a new system

Tracking contributions without adding a new system
Some contribution areas don't have contribution systems or their systems are not eligible for Baloo integration because they track a low volume of contributors (such as Creative, Legal and Policy). Check Ad-Hoc Baloo solution for an alternative way to track contributions in those areas

Several steps are needed in order to add a new system in Baloo. They can be broken down to two different sections. Porting historical data, and implementing live pushes

Adding Historical Data

Once a system is identified holding historical contribution data, the following should be done to import those to Baloo. Note that this should be an one off procedure given that live-pushes are also implemented in a system.

1. Identify *what* do you want to track in your system. Write down all possible contributions that are happening in your area.
2. Talk to your principal webdev or technical people, on how this info is stored in your database.
3. Tweak/amend the Contribution Schema to fit your tracking needs. (Contact Pierros or join Systems and Data WG meeting for help)
4. Open a new bug on Mozilla Metrics::Data/Backend Reports describing the location of the data and the schema/mapping your are about to use
5. Data and BI team will be following up with you on imports

Adding Live Pushes

1. Make sure you follow steps 1,2 and 3 above.
2. Coordinate with your principal webdev or technical people on implementing the code on your system side that produces and pushes the JSON contribution tracking payloads.
3. Open a new bug on Mozilla Metrics::Data/Backend Reports so we can track the implementation process.
4. Tryout staging the code using the following

   curl -v -X POST -d '{"test":"hello"}' http://10.22.28.101:8080/submit/baloo/

   If the operation successful we will return 201.
5. Once you are ready for production you may use the production url: https://data.mozilla.com/submit/baloo/

All data is in raw format till this point
If any or both of the above is implemented for your system, you have raw data in our HBASE. Although this is available for crunching, you still need to establish your contribution area conversion points so we can start tracking active contributors

Tracking active contributors

Once one or both of the processes above are complete we now have raw contribution data about your area. Raw data is cool, but we need to start making sense out of them. For that reason we need to establish Conversion Points to understand the type of contributions and identify Active contributors.

1. Have a meeting with your CBT contact person to identify (if not already done) your contribution pathway.
2. Navigate to our Conversion Points wiki page and add/edit your contribution area with all relevant info.
3. Once the conversion points are up to date, open a bug on Mozilla Metrics::Data/Backend Reports so we can create the aggregation scenarios and start crunching the raw data.
4. Magical numbers will flow in, and become available in Tableau (and APIs!) for usage!

Meetings

You can see meeting notes on the master etherpad

Contact

Pierros and Anurag serve as the main contact people at this point. Reach out to them for further info and instructions on how to add your system to Baloo.

You may also find us on #baloo IRC channel in irc.mozilla.org