cpu//threads number :: avoid oversubscription
EricDeveaud opened this issue · 2 comments
EricDeveaud commented
Hello
Hello
parsnp
uses multiprocessing.cpu_count()
to ge the number of available cpus which returns the number of cpu in the machine. But this is not the same as the number of cpu available to the process. For example, you can run in a taskset context or a batch scheduler like slurm.
see:
$ nproc
96
$ taskset -c 1 nproc
1
$ taskset -c 1 python3 -c "import multiprocessing; print(multiprocessing.cpu_count())"
96
I would suggest to use len(os.sched_getaffinity(0)) instead of multiprocessing.cpu_count()
$ python3 -c "import os; print(len(os.sched_getaffinity(0)))"
96
$ taskset -c 1 python3 -c "import os; print(len(os.sched_getaffinity(0)))"
1
regards
Eric
EricDeveaud commented
ok this is just for a warning but it will be a nicer message
bkille commented
Thanks for the tip @EricDeveaud! I'll be sure to keep this in mind for the future as well.