insitro/redun

Docker executor cannot recognize the volumes option.

Opened this issue · 0 comments

I am trying to get a local setup running for debugging purposes. Data will be staged locally to reduce latency. Unfortunately, the volumes option for the docker executor does not appear to work.

Current config

[executors.docker]
type = docker
max_workers = 20
image = ####
scratch = /tmp/scratch
include_aws_env = False
volumes = [["/home", "/home"]]
interactive = False

The get_docker_executor_config does not retrieve volumes currently.

def get_docker_executor_config(config: SectionProxy) -> SectionProxy:
"""
Returns a config for DockerExecutor.
"""
keys = ["image", "scratch", "job_monitor_interval", "vcpus", "gpus", "memory"]
return create_config_section({key: config[key] for key in keys if key in config})

It may be helpful to throw an error for additional options specified here that cannot be used with the Docker executor.

Additionally, I wonder whether mounting the workflow working directory would be beneficial?

self._default_job_options: Dict[str, Any] = {
"vcpus": config.getint("vcpus", fallback=1),
"gpus": config.getint("gpus", fallback=0),
"memory": config.getint("memory", fallback=4),
"shared_memory": config.getint("shared_memory", fallback=None),
"interactive": config.getboolean("interactive", fallback=False),
"include_aws_env": config.getboolean("include_aws_env", fallback=True),
}