libffcv/ffcv

Default num_workers is incompatible with SLURM

aldakata opened this issue · 0 comments

I encountered a bug in the code. If I run

Loader(path,
            batch_size=1,
            order=OrderOption.SEQUENTIAL,
            pipelines=PIPELINES)

I get

ValueError: The number of threads must be between 1 and 2

Why?
After this request,
numba.config.NUMBA_DEFAULT_NUM_THREADS
The number of usable CPU cores on the system (as determined by len(os.sched_getaffinity(0)), if supported by the OS, or multiprocessing.cpu_count() if not). This is the default value for numba.config.NUMBA_NUM_THREADS unless the NUMBA_NUM_THREADS environment variable is set.

BUT, here the default number of workers is set as multiprocessing.cpu_count(). In the HPC I am working on multiprocessing.cpu_count() = 64 but len(os.sched_getaffinity(0)=2.