Auto-tools/Projects/Pulse

Mozilla Pulse

http://pulse.mozilla.org

Mozilla currently has a ton of different systems that are inter-connected via polling, screen scraping, email, and other brittle methods. To make their lives easier community members often build tools on top of this house of cards, adding yet another level of scraping and polling. Many systems don't even export important data for others to scrape and use, preventing better tools from being written.

The goal of "Mozilla Pulse" is to eliminate polling and add visibility into all aspects of Mozilla and its systems. This allows more robust, dynamic, and informative tools.

File bugs: https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse

Technology used

• The message broker used is RabbitMQ.
• Protocol used to talk to the broker is AMQP.
• Messages are sent by default in JSON, though if you know all the producers and consumers will be in python you can change it to pickle.
• For python, the underlying library currently used to talk AMQP is Carrot.
• Celery is used to run the scrapers that feed data into the broker.
• pubsub plugins should be installed soon
• The STOMP plugin is live, but needs to be documented.
  • Need to document how to connect via STOMP to orbit so that you can get push messages via javascript in the browser.

Task List

• Currently a security hole - anyone can post anything
• Carrot module is currently unmaintained. Any code using Carrot (mozillapulse) should be updated to use Kombu instead.
• Currently all exchanges on one vhost
  • Partition Pulse into multiple vhosts based on function. This is easier to lock down as well since permissions are based on vhost.
• need more vhosts with password accounts
• Enable the use of SSL certificates for all clients.
  • Get IT to provide and maintain the certificates similar to other Mozilla sites.
• mozillapulse currently does not work with more than one vhost so needs to be updated.
• mozillapulse needs to be able to handle network fluctations better.
• Move to buildbot to own vhost
• Pulsetranslator currently uses a durable queue
• Find out if having a staging or development environment for Pulse is possible or even needed.
  • May be able to accomplish this with setting up testing vhosts.
• Look into what would be needed to update RabbitMQ to latest 3.x version to pull in

enhancements/bugfixes.

Admin Procedures

• When a queue becomes stuck, you can use the Admin UI to kill it. Try to ping the queue owner first before killing if possible.
  • More than half of the queues are QA related (whimboo)
• pulsetranslator service is currently running on flyingtanks and may need to be reset from time to time.