does not work in 3 gpus
yqtianust opened this issue · 6 comments
It works well in 2 gpus. However, if I am using 3 gpus, it seems the 3rd job in the queue will not be executed. It shows the 3rd one is running but the text under Output is (...)
BTW, should I use the -S option before any other command?
Well it works. Just need to wait for a relative longer time for allocating etc.
Thanks for the tool.
It will be good if we can use floating number for option G. I want to make my GPU can always occupied by two jobs.
Hi @yqtianust. Thanks for reaching out. I just wanna elaborate a bit more about your experiences.
First, the slow allocation will happen when jobs are queued too fast (for e.g., when jobs are queued with a script) and no matter how many GPUs you use. And as you noticed, the jobs will be executed nevertheless, but the waiting time is quite long sometimes. One way to avoid this is give it some time between each job queue.
should I use the -S option before any other command?
No you don't have to. You can use -S when you start Task Spooler for the first time (for e.g., when you first boot your pc or after you use -K). Or in case you have to share the server with other people, you can set the number of slots to be smaller than the number of GPUs. It's totally up to your use cases.
It will be good if we can use floating number for option G. I want to make my GPU can always occupied by two jobs.
The GPU support is inspired by SLURM and by default, SLURM does not support multiple jobs per GPU (https://slurm.schedmd.com/gres.html). But I understand sometimes there are powerful GPUs and one job uses only a fraction of their memory/usage. One quick fix might be adding a flag to choose GPUs manually. I will look into it in a future version.
Thanks for the elaboration.
Hi @yqtianust. Just FYI, this issue has been fixed following the latest release. There will only be a short delay (30s by default) between two immediately queued GPU jobs.
Well it works. Just need to wait for a relative longer time for allocating etc.
Thanks for the tool.
It will be good if we can use floating number for option G. I want to make my GPU can always occupied by two jobs.
And you can use -g to pass directly GPU IDs as well.
Thanks! I will have a try!