/outsource

easily bid on and manage aws spot instances

Primary LanguagePython

outsource makes it easy to bid on and use AWS Spot Instances, which provide short-term computing power at a much cheaper rate.

Specify your AWS credentials in ~/.aws/credentials:

[default]
aws_access_key_id = YOURACCESSKEY
aws_secret_access_key = YOURSECRETKEY

[another_profile]
aws_access_key_id = YOURACCESSKEY
aws_secret_access_key = YOURSECRETKEY

Then create a config at config.yaml with the following (filled with whatever values you prefer, of course):

region: us-east-1
profile: another_profile
ami: ami-a0c499c8
key_name: my_key_pair

If it will be consistent across your usage.

Then you have a few options:

# Create a spot instance request
$ outsource request <name> <instance type> [--user_data=<path to user data script>] [--ami=<image id>]
$ outsource request doc2vec r3.2xlarge --user_data=user_data.sh
# Then you can do:
$ ssh -i my_key_pair.pem ubuntu@<instance's public ip>
# Try `ec2-user` if `ubuntu` doesn't work

# Cancel a spot instance request
$ outsource cancel <request id>
$ outsource cancel sir-02baen2k

# Terminate a spot instance
$ outsource terminate <name>
$ outsource terminate doc2vec

# List spot instances and requests
$ outsource ls

You can refer to the examples folder for some example user_data scripts.

What outsource does for you:

  • it will look at current bids for your requests instance type in your region, list out their min/max/mean, and suggest a bid price.
  • it will create a security group with SSH access for your local machine (i.e. the machine making the spot instance request). It will also clean up this security group when you terminate the instance.
  • it will tell you the public IP of the spot instance when it's ready.

Reference: