ReleaseEngineering/How To/Manage spot AMIs
Something wrong with spot AMIs
Spot AMIs are generated daily (almost) from scratch daily by aws_create_instance.py
The scripts uses base AMIs generated by aws_create_ami.py. These are generated manually and contain only base system. Particular base AMIs used to generate final spot AMIs are listed in the instance configs (e.g. bld-linux64).
The script takes a base AMI, puppetizes it, and cleans up some files. Spot AMIs use cloud-init to bootstrap their hostnames specified it instance specific user-data.
If something goes wrong, we can delete bad AMIs and automation will be using the previous ones.
To find AMI a particular instance uses you can run the following:
ssh cltbld@instnace_ip curl http://169.254.169.254/latest/meta-data/ami-id
Verify that it matches latest AMIs in get_spot_amis.py output:
# example output $ python scripts/get_spot_amis.py bld-linux64, us-east-1: ami-7af30012 try-linux64, us-east-1: ami-208c7f48 tst-linux64, us-east-1: ami-fa8f7c92 tst-linux32, us-east-1: ami-188f7c70 bld-linux64, us-west-2: ami-3f46340f try-linux64, us-west-2: ami-23473513 tst-linux64, us-west-2: ami-954032a5 tst-linux32, us-west-2: ami-e74032d7
To delete instances generated by those AMIs use the following script