Always out of memory
Closed this issue · 9 comments
Thank you for sharing codes and I have tried your code in my own data. I found that really easy to run out of my GPU memory. Should any downsampling being done? My GPU is 2080Ti with 11G memory and our data is in size of z512512.
Hello ChenWWWeixiang,
Yes, you need to downsample your data in the z-direction. As is pointed out in the Appendix section of our paper, we first resampled our data to the same spacing and then downsampled them by a factor of 5x.
The GPU memory consumption is determined by the maximum z size in your dataset. To make sure that you don't run out of GPU memory, you should check the maximum z-size. For your reference, the average number of slices of the preprocessed data is about 45 and the maximum z size is 147 in our dataset and the GPU size is 16GB. If you still run out of memory, you should probably increase the downsampling factor.
I can't find Appendix in the online paper. Possible to share here/somewhere? Many Thanks!
I can't find Appendix in the online paper. Possible to share here/somewhere? Many Thanks!
Hello jonathan-chan-etrovub,
The Appendix can be can found via: https://pubs.rsna.org/doi/suppl/10.1148/radiol.2020200905/suppl_file/ry_200905_supp_in%20press.pdf
Hi, thanks! Find it.
Another specific question.
In the appendix, it says that 'The weights of the ResNet50 (3) backbone of the COVNet was first initialized to values optimized on the ImageNet database'. Do we need to process the ImageNet database ourselves?
Another specific question.
In the appendix, it says that 'The weights of the ResNet50 (3) backbone of the COVNet was first initialized to values optimized on the ImageNet database'. Do we need to process the ImageNet database ourselves?
No, you don't have to do anything. The ResNet50 backbone is already initialized in the model definition part. Pytorch will automatically download the pretrained weights and automatically use it to initialize the model.
OK, model = models.resnet50(pretrained=True). This part is clear. Thanks.
Sorry for bothering: Does the model do anything even before fine-tuning? In another words, right after the initialization, and before any activate training, can the model already do something if we input a testing case? Probably not ...
OK, model = models.resnet50(pretrained=True). This part is clear. Thanks.
Sorry for bothering: Does the model do anything even before fine-tuning? In another words, right after the initialization, and before any activate training, can the model already do something if we input a testing case? Probably not ...
No, because the last layers are randomly initialized.
How long did it take to train the model with V100 GPU? What was the batch size? Did you use fp16 or fp32?