Balrog: Difference between revisions

Balrog (view source)

Revision as of 18:06, 9 April 2018

34 bytes added , 9 April 2018

→‎ELB Logs

Bhearsum

canmove, Confirmed users

6,439

edits

@@ Line 57: / Line 57: @@
 == ELB Logs ==
-The production instance of Balrog publishes logs to two different S3 buckets:
+Balrog publishes logs to S3 buckets which are [https://sql.telemetry.mozilla.org available for querying in Redash]. The relevant tables are:
-* The nginx access logs (that contain all of the update requests we receive) are published to '''balrog-us-west-2-elb-logs'''. These logs are very large, and you're unlikely to be able to download them for local querying. The best way to work with them is through Athena.
+* balrog_elb_logs_aus{3,4,5} - These tables contain update request records sourced from the ELB logs of the named domain (eg: aus5). If you're looking to do ad-hoc queries of update request (eg: estimate how many users are on a particular version or channel), the balrog_elb_logs_aus5 is probably what you want to query.
-* The rest of the logs are published to '''net-mozaws-prod-us-west-2-logging-balrog''', in the "firehose/s3" directory. Within that there are subdirectories for different parts of Balrog:
+* balrog_elb_logs_aus_api - This table contains request logs for the aus-api.mozilla.org domain
-** balrog.admin.syslog.admin contains the admin wsgi app output.
+* log_balrog_admin_nginx_access - This table contains access logs for the admin app sourced from nginx access logs.
-** balrog.admin.nginx.{access,error} contain the admin access & error logs from nginx. The access logs are generally a subset of the wsgi app output (which logs requests with a bit of extra detail).
+* log_balrog_admin_nginx_error - This table contains error logs for the admin app sourced from nginx error logs.
-** balrog.admin.syslog.agent contains the agent app output.
+* log_balrog_admin_syslog_admin_fixed - This table contains syslog output from the admin app's Docker container.
-** balrog.admin.syslog.cron contains cronjob output (eg: the history cleanup and production database dump)
+* log_balrog_admin_syslog_agent - This table contains syslog output from the agent's Docker container.
-** balrog.web.syslog.web contains the public wsgi app output. Note that this app does _not_ log requests, so this is largely warning/exception output. If you care about requests to the public app, use the nginx access logs (see above).
+* log_balrog_web_syslog_web_fixed - This table contains syslog output from the public app's Docker containers.
+Redash should show you the table schemas in the pane on the left. If not, you can inspect with them with "describe $table".
 == Backups ==