earthlab/rslurm

local_slurm_array on Windows

Opened this issue · 0 comments

I'm finding the rslurm package very useful for making the workflow smoother for using HPC with R - thanks for developing it.

I've been developing submission scripts locally to make sure they are working before sending them to the server, but I found that local_slurm_array didn't work on Windows. I think this was because it relied on bash shell syntax to set the SLURM_ARRAY_TASK_ID environment variable, that was not understood by the Windows command line.

I think this can be made system-independent by using the R functions Sys.setenv and Sys.unsetenv. An adaptation of this function that works for me is

local_slurm_array_portable <- function (slr_job, rscript_path = NULL) 
{
    if (!inherits(slr_job, "slurm_job")) {
        stop("argument of local_slurm_array should be a slurm_job")
    }
    if (is.null(rscript_path)) {
        rscript_path <- file.path(R.home("bin"), "Rscript")
    }
    slurm_run_path <- paste0("_rslurm_", slr_job$jobname)
    wd <- getwd()
    setwd(slurm_run_path)
    for (i in 1:slr_job$nodes - 1) {
        Sys.setenv(SLURM_ARRAY_TASK_ID=i)
        system(paste0(rscript_path, " --vanilla slurm_run.R"))
    }
    Sys.unsetenv("SLURM_ARRAY_TASK_ID") 
    setwd(wd)
    return(slr_job)
}