CIDuty/How To/Troubleshoot AWS

From MozillaWiki
< CIDuty‎ | How To
Revision as of 20:00, 23 October 2018 by Zsolt (talk | contribs) (small typo)
Jump to navigation Jump to search

Sometimes AWS spins up bad instances. Usually sheriffs notify ciduty about these but if you see one escalate to ciduty in #ci. A job may appear as failed if the instance it was running on disappears. Spot instances can disappear when they are outbid.

Bad Instances

To understand if a job failure is caused by a spot instance or not it's best to first understand the various ways a task can be resolved. See this page for more information.

When AWS spins up a bad instances (usually identified by the fact that it fails every job), find it in the worker explorer of AWS Provisioner and terminate it, AWS will spin up a new one. You can do this even while a task is running due to the built in mechanism for retrying jobs. To further understand the interaction between the queue and a worker, check out the official docs.

Retrieved from "https://wiki.allizom.org/index.php?title=CIDuty/How_To/Troubleshoot_AWS&oldid=1202831"