Auto-tools/Projects/Pulse/PulseGuardian: Difference between revisions

Jump to navigation Jump to search

Auto-tools/Projects/Pulse/PulseGuardian (view source)

Revision as of 21:54, 30 April 2014

153 bytes removed ,  30 April 2014
m
PeerGuardian -> PulseGuardian, and added implementation details
m (PeerGuardian -> PulseGuardian, and added implementation details)
Line 1: Line 1:
= Team =
= Team =


* mcote, dkl, ahmed
* mcote, dkl, akachkach


= Problem =
= Problem =
Line 24: Line 24:
The second part is a process that polls RabbitMQ, looking for queues above a set length (WARN_QUEUE_SIZE).  If the queue belongs to a user with a properly formatted username (i.e. an email address), a warning email is sent containing the queue name and current queue length.  After a second threshold is reached (DEL_QUEUE_SIZE), the queue is deleted, and another email is sent.  If the username is not a proper email address (e.g. the public user), the queue is silently deleted when DEL_QUEUE_SIZE is reached (no action is performed at WARN_QUEUE_SIZE).
The second part is a process that polls RabbitMQ, looking for queues above a set length (WARN_QUEUE_SIZE).  If the queue belongs to a user with a properly formatted username (i.e. an email address), a warning email is sent containing the queue name and current queue length.  After a second threshold is reached (DEL_QUEUE_SIZE), the queue is deleted, and another email is sent.  If the username is not a proper email address (e.g. the public user), the queue is silently deleted when DEL_QUEUE_SIZE is reached (no action is performed at WARN_QUEUE_SIZE).


We can also, optionally, add a threshold between WARN_QUEUE_SIZE and DEL_QUEUE_SIZE, call it ARCHIVE_QUEUE_SIZE, at which point PeerGuardian will start to consume messages from the queue and archive them to disk.  This is advantageous because RabbitMQ keeps all queues in memory, so one rogue queue can eventually take down RabbitMQ.  If the queue size falls below ARCHIVE_QUEUE_SIZE, presumably due to the client application resuming, no new messages will be archived unless ARCHIVE_QUEUE_SIZE is exceeded again.  When MAX_ARCHIVE_SIZE messages are archived, messages are no longer consumed by PeerGuardian and thus, unless archived messages are consumed by the client, the queue will continue to grow until DEL_QUEUE_SIZE is hit and the queue deleted, as above.
We can also, optionally, add a threshold between WARN_QUEUE_SIZE and DEL_QUEUE_SIZE, call it ARCHIVE_QUEUE_SIZE, at which point PulseGuardian will start to consume messages from the queue and archive them to disk.  This is advantageous because RabbitMQ keeps all queues in memory, so one rogue queue can eventually take down RabbitMQ.  If the queue size falls below ARCHIVE_QUEUE_SIZE, presumably due to the client application resuming, no new messages will be archived unless ARCHIVE_QUEUE_SIZE is exceeded again.  When MAX_ARCHIVE_SIZE messages are archived, messages are no longer consumed by PulseGuardian and thus, unless archived messages are consumed by the client, the queue will continue to grow until DEL_QUEUE_SIZE is hit and the queue deleted, as above.


We'll have to think through this feature a bit to determine the implications of a client trying to consume while PeerGuardian is also consuming them (or trying to).
We'll have to think through this feature a bit to determine the implications of a client trying to consume while PulseGuardian is also consuming them (or trying to).


= Implementation =
= Implementation =
Django is probably overkill; we can try templeton (https://github.com/markrcote/templeton, common layout & functions based on web.py) or some other simple server.  We'll need a database just to hold pending validations (we'll need to prune those every so often).
PulseGuardian uses Flask for the user management app and sqlalchemy + mysql to store user data.


Communication with RabbitMQ should be via the management plugin's REST API.
Communication with RabbitMQ is done via the rabbitmq management plugin's REST API.
Akachkach
4

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/971703"

Navigation menu