Sheriffing/How To/Retrigger Jobs

Sometimes builds and tests need to be retriggered. For some classes of automation/infrastructure failures, this happens automatically and the job is marked in the Treeherder UI as dark blue. For other cases, or if you're doing investigative work e.g. testing for an intermittent failure, you'll need to retrigger the job manually.

Manual retriggers

• Select a job result in Treeherder and click on it.
• This will display a results pane in the left bottom corner with information like Job, Machine, Task, etc.
• To retrigger this job/test, click on the circular arrow with the mouseover text of "Repeat the selected job" at the top of the results pane and the job will be retriggered. You can accomplish the same thing by simply pressing "r" in the results pane when you are logged in to Treeherder.

Retriggering PGO Builds

With the recent switch to Taskcluster-based builds, every push to inbound/autoland now automatically schedules PGO builds for Linux and Windows, so they can be retriggered manually as needed within Treeherder, just like any other failed job.

Release/ESR52 still need to use self-serve to manually schedule PGO builds. Click the "BuildAPI" link on the desired push in Treeherder, copy the push's revision, paste it into the "Create new PGO builds on <tree>" textbox, and click "Submit" to request them.

Retriggering Nightly Builds

As of Jul 26, 2017 nightly builds for all platforms on m-c are now running in taskcluster so the instructions on how to respin them have been updated for these platforms See https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Other_Duties#Nightlies

CAVEAT: there are implications to triggering too many Nightly builds in a single day or in quick succession. Please talk with a sheriff first before retriggering Nightly builds.

Release/ESR52 still need to use self-serve to manually schedule Nightly builds. Click the "BuildAPI" link on the desired push in Treeherder, copy the push's revision, paste it into the "Create new nightly builds on <tree>" textbox, and click "Submit" to request them.

How to bulk retrigger build bustages a push at a time

Please note that this will run all failed jobs again, not only build bustages!

Prerequisites:

Step 1: Run pip install taskcluster to install a taskcluster component from pip install taskcluster

      If you get the message "The program 'pip' is currently not installed" then you have to install it by running:
           a.  sudo apt install -y python-pip 
           b.  when python-pip install is completed, run pip install taskcluster 

Step 2: Save the file and make it executable by running the command: sudo wget https://hg.mozilla.org/build/braindump/raw-file/default/taskcluster/tc-filter.py -P /usr/local/sbin/ && chmod +x /usr/local/sbin/tc-filter.py

Step 3: Re-sign in with the taskcluster tool if you were already signed in

How to use:

1. sign in with the taskcluster tool ( eval $(taskcluster signin) | if you were not already signed in)
2. run tc-filter.py --state failed --action rerun --graph-id geckoDecisionTaskTaskId

Note: Replace geckoDecisionTaskTaskId with the task id being shown on the bottom left when you click on the gecko decision task for the push with the failures.

Rerunning build bustages

How to install taskcluster CLI

This tool is needed in order to retrigger some build jobs, especially nightly builds. Download Taskcluster CLI on Ubuntu from https://github.com/taskcluster/taskcluster-cli

From your /home/user folder (or the location where mozilla-unified is stored), run the following commands:

1. sudo wget https://index.taskcluster.net/v1/task/project.taskcluster.taskcluster-cli.latest/artifacts/public/linux-amd64/taskcluster -P /usr/local/sbin/
2. sudo chmod +x /usr/local/sbin/taskcluster

The tool is now installed and made executable in /usr/local/sbin/.

How to use Taskcluster CLI

1. From the terminal, run the command: eval $(taskcluster signin). This tool will only work as long as the terminal remains open.
2. When the browser page opens, login using LDAP
3. Click Create a new clientId and go to the end of the page, then click Create Client.
4. Wait a few seconds, then close the browser.
5. In the console, the following message should appear: Credentials output as environment variables.
6. Run taskcluster task rerun TASK_ID (take the TASK_ID from the job summary – go to Treeherder, click the job and on the left side of the window you have Task)
7. After following these steps, the console should output either running or pending.