Default num_workers is incompatible with SLURM
aldakata opened this issue · 0 comments
aldakata commented
I encountered a bug in the code. If I run
Loader(path,
batch_size=1,
order=OrderOption.SEQUENTIAL,
pipelines=PIPELINES)
I get
ValueError: The number of threads must be between 1 and 2
Why?
After this request,
numba.config.NUMBA_DEFAULT_NUM_THREADS
The number of usable CPU cores on the system (as determined by len(os.sched_getaffinity(0)), if supported by the OS, or multiprocessing.cpu_count() if not). This is the default value for numba.config.NUMBA_NUM_THREADS unless the NUMBA_NUM_THREADS environment variable is set.
BUT, here the default number of workers is set as multiprocessing.cpu_count(). In the HPC I am working on multiprocessing.cpu_count() = 64 but len(os.sched_getaffinity(0)=2.