CloudServices/Sagrada/Metlog: Difference between revisions

; generator : The generator portion of the system is the actual service application that is generating the data that is to be sent into the system.  We will provide libraries (described below) that app authors can use to easily plug in.  The libraries will take messages generated by the applications, serialize them, and then send them out to the listener using ZeroMQ.  The metrics generating apps that need to be supported initially are based on the following platforms:

; router : The router is what will be listening for the messages sent out by the provided libraries.  It will deserialize these messages and examine the metadata to determine the appropriate back end(s) to which each message should be delivered.  The format and protocol for delivering these messages to the endpoints will vary from back end to back end.  We plan on using [http://logstash.net/ logstash] as the message router, because it is already planned to be installed on every services server machine, and it is built specifically for this type of event-based messager routing.

* [https://github.com/dcramer/django-sentry Sentry]: '''(Phase 2)''' Sentry is an exception logging infrastructure that provides useful debugging tools to service app developers.  Sentry is not yet planned on being provided by any Mozilla operations team, using it would require buy-in from and coordination with a Mozilla internal service provider (probably the Services Ops team).

* timestamp: Time at which the message is generated.

* logger: String token identifying the message generator, such as the name of the service application in question.

* severity: Numerical code from 0-7 indicating the severity of the message, as defined by [https://tools.ietf.org/html/rfc5424 RFC 5424].

* message: Message text payload.

* metadata: Arbitrary set of key/value pairs that indicates the type of message that is being generated and includes any additional data that may be useful for back end reporting or analysis.

We will provide a "metlog" library that will both ease generation of these messages and that will handle packaging them up and delivering them (via ZeroMQ) into the message processing infrastructure.  Implementations of this library will likely be available in both Python and Javascript, but the Python library will be available first and this document will, for now, only describe the Python API.  The Javascript API will be similar, modulo syntactic sugar that is available in Python but not in JS (e.g. decorators, context managers), and will be documented in detail in the future.  The proposed Python API is as follows:

; '''MetlogClient(host, port, logger="", severity=6)''' : Primary metlog client class which can accept metlog messages and will deliver them to the message processor listening at the specified ''host'' and ''port''.  The provided ''logger'' and ''severity'' values will be used by default for all subsequent ''metlog'' method calls which do not explicitly pass other values.

; '''MetlogClient.set_message_flavor(flavor_name, metadata)''' : The metadata for a given message can be used to label and categorize that message. This method expects a string value ''flavor_name'' and a dictionary ''metadata''. The flavor name value can be passed in as a ''flavor'' to subsequent ''metlog'' calls as shorthand for including the specified metadata in the outgoing message.

; '''MetlogClient.metlog(timestamp=None, logger=None, severity=None, message="", metadata=None, flavors=None)''' : Sends a single log message to the previously specified metlog listener. Most of the arguments correspond to the message fields described above. None of them are strictly required, but most of them will be populated by reasonable defaults if they aren't provided:

* ''metadata'': Defaults to an empty dictionary

; '''MetlogClient.timer(name, timestamp=None, logger=None, severity=None, metadata=None, flavors=None, rate=1)''' : Can be used as either a context manager or a decorator.  Will calculate the time required to execute the enclosed code, and will generate and send a metlog message containing the timing information upon completion.  ''name'' is a required string label for the timer that will be added to the message metadata.  ''rate'' represents what fraction of these invocations should actually be timed, so a value of 0.3 would mean that the code would be timed and the results sent off approximately 30% of the time it was executed.

; '''MetlogClient.incr(name, timestamp=None, logger=None, severity=None, metadata=None, flavors=None)''' : Sends an "increment counter" message to metlog.  ''name'' is a required string label for the counter that will be added to the message metadata.