cloudyr/aws.ec2

Is it possible to create a "spot" instances

Closed this issue · 7 comments

dselivanov commented

Is it possible to create a "spot" instances from run_instances ?

leeper commented

The EC2 API has an endpoint for it, but I haven't implemented it yet. I would gladly welcome a pull request implementing it!

  • 👍1
dselivanov commented

@leeper thx! will take a look.

peterfoley commented

We currently kick out to the aws cli tools for some requests that aren't implemented here yet. Specific ones that I see in our code are request-spot-instances, describe-spot-instance-requests, cancel-spot-instance-requests, describe-spot-price-history, so those 4 (or at least the first 3) seem like the minimal set required for managing spot requests.

I'd be happy to collaborate on getting those working within aws.ec2 itself.

leeper commented

Would be great to have this. If you can at least outline how you're currently using those commands, it should be pretty simple to incorporate this into the package as a set of related functions.

peterfoley commented

Our basic workflow for spinning up the machines looks like:

  • construct spot request json file for the cli tool (not needed for query API)
    • that contains stuff like ImageId, KeyName, SecurityGroupIds, UserData, InstanceType, SubnetId, Monitoring, and optionally IamInstanceProfile
  • place the spot request using aws ec2 request-spot-instances
    • here we also specify the spot price, instance count, expiration, and path to the json file
    • it returns us json describing a set of SpotInstanceRequests (ex. if you request 5 instances, SpotInstanceRequests will be a length-5 array)
  • we poll aws ec2 describe-spot-instance-requests with the request ID's that haven't started yet. As requests are fulfilled we store the instance id's that were started by each request.
  • once the instances start machines or the requests time out, we cancel the requests with aws ec2 cancel-spot-instance-requests so they don't count against our limits (cancelling the request does not terminate the machine that the request started)
peterfoley commented

Recent API changes have made this much easier -- it's just a matter of run_instances adding additional parameters in query before calling ec2HTTP. ec2HTTP call also has to be passed version="2016-11-15", because earlier versions don't accept InstanceMarketOptions. I'm finalizing a patch that I'll be submitting shortly.

leeper commented

I think this was fully closed by #34. If not, I'll reopen.