hustvl/4DGaussians

RuntimeError: CUDA error: initialization error for "CMU Panoptic Dataset"

Closed this issue · 3 comments

Cranjis-McB commented

First of all, Great work. I was running the code for CMU Panoptic Dataset but got the CUDA Runtime Error. Upon Debugging I figured out after loading data in Dataloader here. It throws up CUDA Runtime Error in this line. Attaching the logs below for better understanding. Thanks in advance.

Logs:

RuntimeError: CUDA error: initialization error
Traceback (most recent call last):
File "/home/vikarm/anaconda3/envs/Gaussians4D/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
obj = _ForkingPickler.dumps(obj)
File "/home/vikarm/anaconda3/envs/Gaussians4D/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "/home/vikarm/anaconda3/envs/Gaussians4D/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 259, in reduce_tensor
event_sync_required) = storage.share_cuda()
File "/home/vikarm/anaconda3/envs/Gaussians4D/lib/python3.7/site-packages/torch/storage.py", line 800, in share_cuda
return self._storage.share_cuda(*args, **kwargs)

Cranjis-McB commented

I figured out the workaround. Thanks.

BruceKnife commented

I figured out the workaround. Thanks.

Hello！How to solve this problem? I have same problem.

Cranjis-McB commented

Hi @BruceKnife,

Sorry for the late reply. I believe I was encountering the error because the readPanopticMeta function loads all the images at once, which exceeded the available GPU memory. To address this, I created a dataset class similar to the one in multipleview_dataset.py, which loads the data dynamically as needed during runtime.