ReleaseEngineering/Applications/Runner

Runner is an task management system which works like an init system that runs in a continuous loop. Users may define custom environment variables, specify failure behaviors, and set up dependencies between tasks. Runner also makes it easy to track the progress of systems through their task graph via logging hooks.

For usage information see: https://github.com/mozilla/build-runner

Mozilla Deployment

Runner is deployed via puppet and runs on each platform as a startup service or task. During installation runner’s configuration is generated along with the platform specific tasks it should run: http://hg.mozilla.org/build/puppet/file/1185781bb6c1/modules/runner/templates/tasks/

Adding New Tasks

Runner tasks are simply executable scripts of any kind, as such, you may store them in the runner puppet module as either a template or a file.

A function exists which will handle the details of placing task scripts and setting their permissions properly, for examples see any of the existing tasks at: http://hg.mozilla.org/build/puppet/file/1185781bb6c1/modules/runner/manifests/tasks

Finally, to enable a task add include runner::tasks::my_new_task to a puppet manifest.

Logging

Runner includes a task hook, which allows users to specify a command that runner will call before and after each task: passing along a JSON blob of context as an argument. A script is being used to pass runner’s state to influxdb at this time: http://hg.mozilla.org/build/puppet/file/953387c94f60/modules/runner/files/influxdb_hook.py

The influxdb table is named runner_tasks, and it's schema is as follows:

   {
       "columns": [
           "time",
           "sequence_number",
           "aws_instance_id",
           "task", # name of the currently running task
           "aws_instance_type",
           "placement/availability_zone",
           "result", # corresponds to tasks exit code (or RUNNING if it hasn't yet finished)
           "hostname",
           "aws_ami_id",
           "platform", # linux/mac/windows
           "max_retries", # how many retries before the machine reboots
           "try_num", # what retry number is this task on
       ],
   }

For docs about InfluxDB's query language see: http://influxdb.com/docs/v0.8/api/query_language.html

Runner’s regular output can be found on Linux hosts at /var/log/runner.log, on OS X hosts at /var/tmp/runner.out and /var/tmp/runner.err, and on Win32 hosts at C:/tmp/runner.log