Training freezes and gets stuck

[henrypearce4D]

Training freezes and gets stuck, happens at low iterations

Reading camera 176/176 [25/06 11:43:34]
Generating random point cloud (100000)... [25/06 11:43:34]
Loading Training Cameras [25/06 11:43:35]
Loading Test Cameras [25/06 11:44:43]
Number of points at initialisation :  100000 [25/06 11:44:52]
Training progress:   3%|█▎                                         | 900/30000 [00:30<05:29, 88.41it/s, Loss=0.4608608]

[mbernier-arcturus]

Check your GPU memory, in some cases during training it goes haywire and GPU memory bust. I got this error while trying different combinations of parameters.

[shakibakh]

Can you please specify the setup and the shape you are trying to run and also your environment?
The code is only tested on Ubuntu, the specifics are now updated in README.

[henrypearce4D]

Can you please specify the setup and the shape you are trying to run and also your environment? The code is only tested on Ubuntu, the specifics are now updated in README.

Hi, this is my conda env running on windows 10 cuda toolkit 11.8

channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - cudatoolkit=11.6
  - plyfile
  - python=3.8
  - pip=22.3.1
  - pytorch=1.12.1
  - torchaudio=0.12.1
  - torchvision=0.13.1
  - tqdm
  - packaging
  - pip:
    - submodules/diff-gaussian-rasterization
    - submodules/simple-knn```

[NeutrinoLiu]

one reason might be the relocation stage, it takes long time if there are a lot of "dead gaussian" in the scene (i suppose, not sure). I recommend lower the max_cap from 1M to some lower values. for me i just use 100K for simple scene. that might help.

[ekhaliul]

I had a similar issue and this fix resolved it for me.

[henrypearce4D]

I had a similar issue and this fix resolved it for me.

Thanks for the suggestion, I already switched to using the implementation in nerfstudio

[henrypearce4D]

hi all, revisiting this it does look related to the max cap value.

I'm using 190 images at 4112 x 3008 res on -r 2 with --data_device cpu flag.

If I increase the --cap_max 1000000 to anything over that then limits are being reached.

In task manager you can see that the GPU memory is still maxing even though im using the data device cpu flag, is this because I'm using the -r flag but not actually using downscaled images?

compared to postshot mcmc, I can set that at 3m max cap and use a downscaled res of 75% so seams more optimised

[shakibakh]

Closing the issue. Please feel free to reopen it if you have any further questions or concerns.