cpu//threads number :: avoid oversubscription

[EricDeveaud]

Hello

Hello

parsnp uses multiprocessing.cpu_count() to ge the number of available cpus which returns the number of cpu in the machine. But this is not the same as the number of cpu available to the process. For example, you can run in a taskset context or a batch scheduler like slurm.

see:

$ nproc
96
$ taskset -c 1 nproc
1
$ taskset -c 1 python3 -c "import multiprocessing; print(multiprocessing.cpu_count())"
96

I would suggest to use len(os.sched_getaffinity(0)) instead of multiprocessing.cpu_count()

$ python3 -c "import os; print(len(os.sched_getaffinity(0)))"
96
$ taskset -c 1 python3 -c "import os; print(len(os.sched_getaffinity(0)))"
1

regards

Eric

[EricDeveaud]

ok this is just for a warning but it will be a nicer message

[bkille]

Thanks for the tip @EricDeveaud! I'll be sure to keep this in mind for the future as well.