Need to increase shm size when using PyTorch DataLoader
MRrollingJerry opened this issue · 1 comments
MRrollingJerry commented
Describe the bug
I was using google could platform. When using dataloader, multiprocessing=True, got thread is killed: bus error because docker limits resources.
Solution
When opening jupyter notebook in the shell, run
sudo nvidia-docker run --shm-size=1g --rm -p 8888:8888 -v /home/$USER:/home/jovyan nvaitc/ai-lab
instead of
sudo nvidia-docker run --rm -p 8888:8888 -v /home/$USER:/home/jovyan nvaitc/ai-lab
The new command assigns 1 GB to the container which allows multiprocessing
tlkh commented
To-do:
- Update documentation