tlkh/ai-lab

Need to increase shm size when using PyTorch DataLoader

MRrollingJerry opened this issue · 1 comments

Describe the bug
I was using google could platform. When using dataloader, multiprocessing=True, got thread is killed: bus error because docker limits resources.

Solution
When opening jupyter notebook in the shell, run
sudo nvidia-docker run --shm-size=1g --rm -p 8888:8888 -v /home/$USER:/home/jovyan nvaitc/ai-lab
instead of
sudo nvidia-docker run --rm -p 8888:8888 -v /home/$USER:/home/jovyan nvaitc/ai-lab

The new command assigns 1 GB to the container which allows multiprocessing

tlkh commented

To-do:

  • Update documentation