NVlabs/SMERF

How much GPU memory is enough for the default config? Nvidia Tesla V100 32GB still OOM when runing it.

Yang-Jianzhang opened this issue · 2 comments

Yang-Jianzhang commented

My device is 8 x Nvidia Tesla V100 32GB.

I try to run the defatult config with command:
python tools/train.py projects/openlanev2/configs/baseline_large_ptsrep_smerf.py
without changing any config or code.

but I still got the OOM error.

I wonder how much GPU memory is enough?

katieluo88 commented

Hi! I was running the models with A100 (40 GB ones, I know, it's a bit strange) so they're a bit larger than your memory. One thing I would suggest is to run the toponet baseline. Their model is more lightweight and (as a plus) yields better results. Run distributed training with:
bash tools/dist_train.sh projects/toponet_openlanev2/configs/toponet_smerf.py 8

This should definitely fit on the V100s that you're using.

  • 👍1
Yang-Jianzhang commented

@katieluo88 Thanks!
I suggest you also provide this more lightweight config in your README, for the people who do not have such advanced devices but interested in this excited project.

Thanks again~