ReleaseEngineering/How To/Request a loaner

From MozillaWiki
Jump to: navigation, search

It is possible for developers to borrow build or test machines to investigate failures that you cannot reproduce locally. A machine can be requested by filing a bug in the Loan Requests component in Bugzilla.

What type of machines do I need?

If you're looking at a result on Treeherder, the pop-up frame at the bottom that contains the job summary and logs also contains the name of the machine that ran the job, e.g.:

Machine name: tst-linux32-spot-1024

To make requesting the correct type of machine easier, here's a mapping of platforms/machine types to bugzilla request templates for that type of machine:

[NOTE]: we do not loan spot machines since they're a bid_for_low_price version of on-demand instances (ec2) and they could go away anytime the spot price become greater than our bid price.

e.g.: if the failure occurred on a 'tst-linux32-spot' machine, you'll want to request a 'tst-linux32-ec2' loaner for debugging.

    • Windows (TaskCluster - these instances can be self service provisioned. no need to raise a bug)
      • gecko-1-b-win2012 Windows Server 2012 r2 - Use for debugging Build issues
      • gecko-t-win10-64 Windows 10 Enterprise, x86_64 - Use for debugging Test issues
      • gecko-t-win10-64-gpu Windows 10 Enterprise, x86_64 - Use for debugging Test issues where a GPU is required
      • gecko-t-win7-32 Windows 7 Enterprise, x86 - Use for debugging Test issues
      • gecko-t-win7-32-gpu Windows 7 Enterprise, x86 - Use for debugging Test issues where a GPU is required

What happens next?

If your loaner request bug has sat without action for more than a day, please ping the person on buildduty in the #releng channel. Once the person on buildduty notices your loaner request bug, they will perform the necessary actions actions to procure you a machine of the chosen type.

When the loaner is ready for you, the original request bug will be assigned back to *YOU*, the developer, for the duration of your use of the loaner. The loan request bug will block a bug for the specific machine you have borrowed.

Accessing your loaner

When the loaner is ready for you, the person on buildduty will contact you out-of-band (usually via email) to give you login crendentials and any specific notes for the loan.


To access all loaners, you require access to the Mozilla VPN, and as such, require Mozilla LDAP credentials. Buildduty will work with IT to get the VPN access setup prior to handing over your requested machine.

Details for accessing the VPN can be found here:

ssh vs. VNC

All loaners *should* be accessible via ssh and VNC. However, VNC access is required to run tests in the same graphical context as run by buildbot (the releng continuous integration framework). Kicking off tests via ssh will give unpredictable (read: wrong) results.

Best practice is to start tests via VNC, and then examine logs, etc via ssh.

Mac can be funky to VNC into. Read the "Tips & Tricks" section below.

Caveat: if you are specifically concerned about performance numbers (i.e. talos), you should avoid touching the loaner at all once the test has begun, i.e. don't be connected via ssh running tail on a log while the test is running, etc.

How do I recreate this failure?

These days, most jobs are run via mozharness scripts which live in-tree alongside their configs. This makes it easier to replicate a given build or test failure by hand.

If you're trying to debug a failure that isn't in mozharness, the log from treeherder will include all of the steps and environment variables you need to setup to replicate the output of a given task.

Can I install X? - root access

Yes, you can.

If there's a debugging tool or something that makes your life easier while working with the loaner, feel free to install it. In most cases, you should have root/Administrator/sudo access to install things as required. We re-image all loaned machines when they come back from developers, so changes will be lost and your machine will return to a known state.

To have root access make sure that you ssh directly as root rather than trying to switch users.

Caveat #1: If you're trying to debug a failure that is happening in buildbot, try to replicate the failure *before* installing extra software of upgrading packages. Every package or configuration option you change makes your results less applicable to the current production environment.

Caveat #2: If you're working with a loaner and *do* need to install software, drivers, service packs, etc. in order to make a build or test work, please keep detailed notes. If these are packages that will need to be deployed to all machines of that type, your detailed notes will make life much easier for releng and relops.

Returning your loaner

When you are finished with the loaner, please comment to that affect in the request bug and resolve it as FIXED. This is the cue to releng that we can recover that loaner.

Pinging the person on buildduty in #releng is appreciated as well.

Tips & Tricks

Machine type-specific notes

Mac OS X

To VNC into the Mac machines can be rather difficult. Try Armen's Lion instructions, and if that doesn't work ask #releng to help you.

Linux test machine

You'll need a newer compiler. If you choose gcc-4.8, you'll need a newer gdb or compile asking for a older version of debug information


To ssh into the Windows machines, you need to specify hmac-md5 as the MAC algorithm. e.g.

ssh -m hmac-md5