ReleaseEngineering/How To/Manage spot AMIs

From MozillaWiki
Jump to: navigation, search


Something wrong with spot AMIs


Spot AMIs are generated daily (almost) from scratch daily by

The scripts uses base AMIs generated by These are generated manually and contain only base system. Particular base AMIs used to generate final spot AMIs are listed in the instance configs (e.g. bld-linux64).

The script takes a base AMI, puppetizes it, and cleans up some files. Spot AMIs use cloud-init to bootstrap their hostnames specified it instance specific user-data.

If something goes wrong, we can delete bad AMIs and automation will be using the previous ones.


To find AMI a particular instance uses you can run the following:

ssh cltbld@instnace_ip curl

Verify that it matches latest AMIs in output:

# example output
$ python scripts/
bld-linux64, us-east-1: ami-7af30012
try-linux64, us-east-1: ami-208c7f48
tst-linux64, us-east-1: ami-fa8f7c92
tst-linux32, us-east-1: ami-188f7c70
bld-linux64, us-west-2: ami-3f46340f
try-linux64, us-west-2: ami-23473513
tst-linux64, us-west-2: ami-954032a5
tst-linux32, us-west-2: ami-e74032d7

Delete the AMIs and the corresponding snapshots using AWS Web Console: us-east-1 us-west-2 then filter by ami name

To delete instances generated by those AMIs use the following script