Baloo
Intro
Project Baloo, is a collaborative effort between the Data and BI Services and Community Building team to create a contribution tracking system for Mozilla.
Status
<smartsheet id="bdc633a5864c43ff9c07893c947e2a29" width="100%" height="400px" />
Roadmap
Q1 2014
- Establish Baloo documentation and wiki pages -- DONE
- Creation of appropriate bagheera entry points -- DONE
- Tryout backporting of historical data from a major contribution system (SuMo?) -- DONE
Q2 2014
- Start production backporting of historical data from all major contribution systems -- ONGOING
| System | Area | Schema Update | Importing historical data | Aggregation Scenarios | On 10x dashboard |
|---|---|---|---|---|---|
| Support.mozilla.org | SUMO | Yes | bug 980385 | bug 1010197 | Done |
| bugzilla.mozilla.org | QA | bug 990667 | bug 1019689 | bug 1010205 | Done |
| reps.mozilla.org | Reps | bug 1010180 | bug 1010183 (25%) | bug 1010207 | |
| Bugzilla and Github | Coding | bug 1010187 | bug 1010190 | bug 1010209 | bug 1025952 |
| developer.mozilla.org | Dev Eng | bug 990180 | bug 1010191 (0%) | bug 1010203 |
| Done | In process | Blocked | No action yet |
- Creation of first dashboard based on 2013 historical data
Q3 2014
| System | Area | Schema Update | Import of data | Building Aggregations | On 10x dashboard |
|---|---|---|---|---|---|
| addons.mozilla.org | AMMO | ||||
| marketplace.mozilla.org | AMMO | ||||
| github.com | WebDev | ||||
| location.services.mozilla.com | Location Services | ||||
| webcompat.com | Web Compatibility | ||||
| moztrap.mozilla.org | QA | 1042101 | |||
| oneanddone.mozilla.org | QA | ||||
| mofointegration DB | Webmaker | Schema | 1047299 | 1047309 | |
| mofointegration DB | Science Lab | Schema | 1047299 | ||
| mofointegration DB | Open News | Schema | |||
| mofointegration DB | MoCo Ad-hoc | Schema | 1047299 | ||
| mofointegration DB | MozFest | Schema | 1047299 |
| Done | In process | Blocked | No action yet |
18 Total; 3 Open (16.67%); 15 Resolved (83.33%); 0 Verified (0%);
- Integrate Baloo for top-10 areas left over from Q2
- Localization
- Location Services
- WebDev
- Web Compatibility
- Continuing development of Ad-hoc contributions tracker to include MoCo areas listed below and pushing data to Baloo
- Legal
- Privacy
- Finance
- User engagement
- Engagement l10n
- User Research
Q4 2014
- 100% coverage of contribution systems across mozilla (including the manual non-automated ones)
Technical Documentation
Baloo is re-using Data and BI Services team technological stack re-purposed to suit the Contributor tracking needs. Have a look on the data flow diagram below to understand the inner workings.
Stack
Schema
Baloo deals with Contribution Data. Contribution Data are defined using the established Schema of the Community Building Team.
Adding a new system
Some contribution areas don't have contribution systems or their systems are not eligible for Baloo integration because they track a low volume of contributors (such as Creative, Legal and Policy). Check Ad-Hoc Baloo solution for an alternative way to track contributions in those areas
Several steps are needed in order to add a new system in Baloo. They can be broken down to two different sections. Porting historical data, and implementing live pushes
Adding Historical Data
Once a system is identified holding historical contribution data, the following should be done to import those to Baloo. Note that this should be an one off procedure given that live-pushes are also implemented in a system.
- Identify *what* do you want to track in your system. Write down all possible contributions that are happening in your area.
- Talk to your principal webdev or technical people, on how this info is stored in your database.
- Tweak/amend the Contribution Schema to fit your tracking needs. (Contact Pierros or join Systems and Data WG meeting for help)
- Open a new bug on Mozilla Metrics::Data/Backend Reports describing the location of the data and the schema/mapping your are about to use
- Data and BI team will be following up with you on imports
Adding Live Pushes
- Make sure you follow steps 1,2 and 3 above.
- Coordinate with your principal webdev or technical people on implementing the code on your system side that produces and pushes the JSON contribution tracking payloads.
- Open a new bug on Mozilla Metrics::Data/Backend Reports so we can track the implementation process.
- Tryout staging the code using the following
curl -v -X POST -d '{"test":"hello"}' http://10.22.28.101:8080/submit/baloo/If the operation successful we will return 201. - Once you are ready for production you may use the production url: https://data.mozilla.com/submit/baloo/
If any or both of the above is implemented for your system, you have raw data in our HBASE. Although this is available for crunching, you still need to establish your contribution area conversion points so we can start tracking active contributors
Tracking active contributors
Once one or both of the processes above are complete we now have raw contribution data about your area. Raw data is cool, but we need to start making sense out of them. For that reason we need to establish Conversion Points to understand the type of contributions and identify Active contributors.
- Have a meeting with your CBT contact person to identify (if not already done) your contribution pathway.
- Navigate to our Conversion Points wiki page and add/edit your contribution area with all relevant info.
- Once the conversion points are up to date, open a bug on Mozilla Metrics::Data/Backend Reports so we can create the aggregation scenarios and start crunching the raw data.
- Magical numbers will flow in, and become available in Tableau (and APIs!) for usage!
Meetings
You can also see meeting notes on the master etherpad
Contact
Pierros and Anurag serve as the main contact people at this point. Reach out to them for further info and instructions on how to add your system to Baloo.
You may also find us on #baloo IRC channel in irc.mozilla.org