Auto-tools/Projects/Pulse
Mozilla Pulse
Mozilla currently has a ton of different systems that are inter-connected via polling, screen scraping, email, and other brittle methods. To make their lives easier community members often build tools on top of this house of cards, adding yet another level of scraping and polling. Many systems don't even export important data for others to scrape and use, preventing better tools from being written.
The goal of "Mozilla Pulse" is to eliminate polling and add visibility into all aspects of Mozilla and its systems. This allows more robust, dynamic, and informative tools.
File bugs: https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse
Status
At the moment, only buildbot (build and test) and simple Bugzilla update messages are being published to Pulse. The other publishers have been disabled for various reasons. We can investigate turning them back on once we have completed some important fixes and features.
Technology used
- The message broker used is RabbitMQ.
- Protocol used to talk to the broker is AMQP.
- Messages are sent by default in JSON, though if you know all the producers and consumers will be in python you can change it to pickle.
- For python, the underlying library currently used to talk AMQP is Kombu.
- Celery is used to run the scrapers that feed data into the broker.
- pubsub plugins should be installed soon
- The STOMP plugin is live, but needs to be documented.
- Need to document how to connect via STOMP to orbit so that you can get push messages via javascript in the browser.
Road Map
Management
- Intelligently handle queues that start filling up.
- See PulseGuardian.
Security
- [DONE] Enable SSL.
- Enable SSL by default in clients.
- Close non-SSL port eventually?
- Partition services into different vhosts, one user with write permissions per vhost. Configure existing shims appropriately.
- For simplicity and to ease upgrades, the vhost should be coded into the mozillapulse publishers and consumers.
- PulseGuardian will have to be updated for this.
- After a grace period following PulseGuardian's launch, remove the "public" user.
- Move to a tighter permission model. New software (an external app or a plugin) may be required to achieve all points.
- Only publishers should be able to declare exchanges.
- Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
- Only the user that created a particular queue should be allowed to consume from it.
Other
- Upgrade RabbitMQ to latest 3.x version (ideally with zero downtime).
- Integrate PulseGuardian with old pulse website.
- Indicate current pulse status (at least just up/down) on website.
- (Maybe) Display published messages on the pulse website (mostly decorative).
- Re-enable hg shim?
- Add git shim?
- Other shims?
Admin Procedures
- When a queue becomes stuck, you can use the Admin UI to kill it. Try to ping the queue owner first before killing if possible.
- More than half of the queues are QA related (whimboo)
- pulsetranslator service, which normalizes buildbot messages, is currently running on pulsetranslator.ateam.phx1.mozilla.com and may need to be reset from time to time.
- logparser service, used by Orange Factor, runs on orangefactor1.dmz.phx1.mozilla.com
More reading
LegNeato wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.
- Mozilla Pulse and RabbitMQ
- Push notifications for Bugzilla
- This idea has been shelved for the time being, both due to security issues and performance. See BMO/ChangeNotificationSystem for an alternative.
- bugzilla-amqp is now buzgilla-push, supports the STOMP protocol
- See above note. Just keeping these links for interest's sake.
- What’s up with Pulse?
- GitHub AMQP integration service hook live!
- Tangentially related to Pulse.
- Lots of Pulse changes, out of prototype mode this week!
- Another update on Pulse