add_objects doesn't save local variables

[asmagen]

Hello,

I'm running rslurm from within a function. The function creates local variables (say a matrix of values) that needs to be read in a slurm job.

I'm getting an error in saving the objects through the add_objects option (variables do not exist). I'm reading here that the variables need to be environment variables and not local ones. But why is that? And how can I bypass that?

Thanks,
Assaf

[pmarchand1]

Hi Assaf,

To clarify, you are running a function (say func1), and within it you're running another function (say func2) in parallel with slurm_apply?

e.g.

func1 <- function(...) {
[....]
obj1 = [...]
func2 <- function(...) {
[...]
}
slurm_apply(func2, params, add_objects = "obj1")
}

Where is func2 defined? I think (though @itcarroll may have a better idea) that it will work if func2 is defined within func1, as in the example above. This is because add_objects looks for objects within the parent environment of the function being passed to slurm_apply. So if add_objects are local to a larger function, the function being parallelized also needs to be local.

[asmagen]

Interesting. Defining the parallelized function to within the running function solves this. But I wonder why is it so? Why would it look for the objects in such a manner?

[itcarroll]

You are familiar with R function environments?

Variables named in add_objects are assumed to live within any of the enclosing environments of the function being passed to the Slurm job. This emulates the usual lexical scoping in R ... at least it is meant to.

If you still need assistance, please re-open this issue with a minimal working example of the problem.