Auto-tools/Projects/Pulse actions

Description

Pulse Actions has two python workers listening to pulse messages from Treeherder in order to take actions for sheriffs (from the TH UI) and based on jobs failing.

In order to disable any of the workers:

• Add --dry-run to the appropiate worker on Procfile
• Push the change to the 'heroku' branch (not master)
• The change will be autodeployed

Info

• Source code: https://github.com/mozilla/pulse_actions
• Running on: Heroku
• Logs on:
  • For corp access -> https://papertrailapp.com/systems/pulse-actions/events
  • For communit access:
    • https://dashboard.heroku.com/apps/pulse-actions/resources
    • Click on PaperTrail icon

Reverting a deployment

• Visit the [1] page.
• Choose the revision you want to roll back to and click "Rollback to here"

Test locally

• Setup your Pulse credentials
  • If you don't have a pulse client visit [2] and create a user
  • export PULSE_USER
  • export PULSE_PW
• We're going to be listening for the pulse stream of the staging version of Treeherder
  • Ask someone in #treeherder to make your account be of a sheriff role (since only sheriffs can trigger actions)
• Start one of the workers which listens to the specified pulse exchanges and routing keys
  • valid topics are the keys in run_time_config.json
  • the ones ending with "stage" will listen to TH's staging version
  • Use --dry-run to avoid actually scheduling anything

python pulse_actions/worker.py --topic-base topic-stage --dry-run

Alerts

• Papertrail alerts sent to armenzg and others - https://papertrailapp.com/alerts
  • Memory issues
  • Worker1: Errors
  • Worker1: crashed
  • Worker2: Errors
  • Worker2: crashed

Accounts

• Papertrail:
  • For employees, you can request a corp account to all Papertrail apps
  • For contributors, speak with armenzg to add you to the Papertrail add-on
• Pulse:
  • Visit pulse guardian and create your credentials

Credentials

For pulse_actions to work we need set these environment variables:

• Buildapi - LDAP credentials (LDAP_PW/LDAP_USER)
• TaskCluster - TaskCluster client credentials (TASKCLUSTER_CLIENT_ID/TASKCLUSTER_ACCESS_TOKEN)
• Pulse - Pulse user credentials (PULSE_PW/PULSE_USER)
• Papertrail - API Token (PAPERTRAIL_API_TOKEN)

To run locally you will need to set all up as env variables. You won't need to add the Papertail token.

Workflow

This workflow was created with Gliffy: http://www.gliffy.com/go/publish/8686299

Support

In case the service is causing issues:

• Point of contact: armenzg
• Team alias address: auto-tools
• Brief description:
  • Pulse_actions listens to various pulse exchanges and takes various scheduling actions through BuildApi.
• What to do in case of issues: Send an email to auto-tools and ask someone to stop it until point of contact can look into it.
• Impact: If you disable pulse_actions:
  • sheriffs will not be able to backfill jobs (and other actions) from Treeherder
    • Re-opening trees would take longer
  • developers/sheriffs would not be able to add missing jobs to pushes

Taskcluster scopes

In the future, we will be using TaskCluster to schedule tasks.

These are the scopes which I have needed in the past:

• queue:create-task:*
• queue:define-task:*
• queue:route:*
• scheduler:create-task-graph
• scheduler:extend-task-graph

Scopes needed because of tasks already defined in-tree:

• docker-worker:cache:*
• docker-worker:capability:*
• docker-worker:image:*