signetlabdei/sem

Using DRMAA clusters for parallel execution of simulations

matbord opened this issue · 2 comments

I would like to use the DRMAA: from the documentation I saw I have to use the method create_runner(script,runner_type=’Auto’,optimized=True,skip_configuration=False,max_parallel_processes=None) and set runner_type=GridRunner.
I don't understand where do I have to create this runner? and once I create it, how can I run all the simulations?

To make it easy, if I have the following basic code, what do I have to change to use the DRMAA?

campaign  = sem.CampaignManager.new(ns_path, script, campaign_dir, overwrite=False, check_repo=False)
paramstot = contains the parameters for my simulation                                         
campaign.run_missing_simulations(paramstot, runs=4)
resultDF  = campaign.get_result_as_dataframe(methodname, labels)

DRMAA support is experimental and wasn't tested very thoroughly. Furthermore, this might also not be the best solution if you are executing SEM from a "login" node that should only be used for submission, as the main process will still consume resources on the login node and possibly raise some sysadmin eyebrows.

I'm working on better support for SLURM, and thinking about dropping DRMAA support - in the meantime, if you want to run SEM on a cluster, you can use the parallel runner as normal, and simply submit your python script to the cluster as a single job. If, for instance, you request 32 CPUs to the cluster, make sure to set max_parallel_processes to 32 and launch your script on the cluster using qsub/sbatch. The process will use all the available CPUs, and the efficiency of resource utilization will be almost the same as with the gridrunner.

ok, I'll try, thankyou very much!