cloudyr/aws.ec2

Is it possible to start a cluster?

Opened this issue · 2 comments

ignacio82 commented

I found this 2014 tutorial about running R jobs on an AWS EC2 cluster. Is it possible to use aws.ec2 to make this process easier? If so, is there an example somewhere?

leeper commented

Not as of yet but that's exactly the kind of functionality I'd like to get implemented. I'll try to get to it soon.

leeper commented

Actually, reading through that tutorial, this should already be possible. All it's doing is the equivalent of calling run_instances() on a master node, that is: (1) fire up a head instance using run_instances() locally, then (2) logging into that instance and doing run_instance() to spawn further machines. The rest of it is just snow::makeCluster().

So that means it might be useful to add some higher-level functions and documentation thereof:

  • run_instances() with a tidier response structure; maybe a new function, initalize_cluster()
  • kill_cluster() that runs stop_instances() and terminate_instances() on all instances created by initialize_cluster()
  • A tutorial showing how to feed this into:
    • snow::makeCluster() and snow::stopCluster()
    • cl <- parallel::makeCluster(c("n1", "n2", "n3")); future::plan(cluster, workers = cl)