CIDuty/How To/Troubleshoot AWS

From MozillaWiki
< CIDuty‎ | How To
Revision as of 19:54, 23 October 2018 by Zsolt (talk | contribs) (changed quarantine to terminate)
Jump to navigation Jump to search

Sometimes AWS spins up bad instances. Usually sheriffs notifies ciduty about these but if you see one escalate to ciduty in #ci. A job may appear as failed if the instance it was running on disappears. Spot instances can disappear when they are outbid.

Bad Instances

To understand if a job failure is caused by a spot instance or not it's best to first understand the various ways a task can be resolved. See this page for more information.

When AWS spins up a bad instances (usually identified by the fact that it fails every job), find it in the worker explorer of AWS Provisioner and terminate it, AWS will spin up a new one. You can do this even while a task is running due to the built in mechanism for retrying jobs. To further understand the interaction between the queue and a worker, check out the official docs.

Retrieved from "https://wiki.allizom.org/index.php?title=CIDuty/How_To/Troubleshoot_AWS&oldid=1202830"