Out of memory with 1 image and Titan X Pascal
Closed this issue · 4 comments
MicaelCarvalho commented
I have only one image in the input folder (style has a different size):
$ identify examples/*/*
examples/input/in1.png PNG 1000x750 1000x750+0+0 8-bit sRGB 970KB 0.000u 0:00.009
examples/segmentation/in1.png PNG 1000x750 1000x750+0+0 8-bit sRGB 4.09KB 0.000u 0:00.000
examples/segmentation/tar1.png PNG 700x393 700x393+0+0 8-bit sRGB 938B 0.000u 0:00.000
examples/style/tar1.png PNG 700x393 700x393+0+0 8-bit sRGB 586KB 0.000u 0:00.000
Segmentation masks are completely black, just like for example in9, and I believe I should have enough GPU horsepower for this:
$ nvidia-smi
Wed Mar 29 11:26:09 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26 Driver Version: 375.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN X (Pascal) Off | 0000:02:00.0 Off | N/A |
| 23% 24C P8 9W / 250W | 0MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 TITAN X (Pascal) Off | 0000:03:00.0 Off | N/A |
| 23% 30C P8 10W / 250W | 0MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 TITAN X (Pascal) Off | 0000:82:00.0 Off | N/A |
| 23% 23C P8 8W / 250W | 0MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 TITAN X (Pascal) Off | 0000:83:00.0 Off | N/A |
| 23% 27C P8 9W / 250W | 0MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
But I get out of memory errors whenever trying to run gen_all.py (with modified variables for 1 image and 1 GPU) or the th script directly:
$ python gen_all.py
working on image pair index = 1
gpu, idx = 0 1
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 574671192
Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
Exp serial: examples/tmp_results
Setting up style layer 2 : relu1_1
Setting up style layer 7 : relu2_1
Setting up style layer 12 : relu3_1
Setting up style layer 21 : relu4_1
Setting up content layer 23 : relu4_2
Setting up style layer 30 : relu5_1
WARNING: Skipping content loss
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-6378/cutorch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
.../torch/install/bin/luajit: .../torch/install/share/lua/5.1/nn/Container.lua:67:
In 4 module of nn.Sequential:
cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-6378/cutorch/lib/THC/generic/THCStorage.cu:66
stack traceback:
[C]: at 0x7f1263b53940
[C]: in function 'cmul'
neuralstyle_seg.lua:465: in function <neuralstyle_seg.lua:456>
[C]: in function 'xpcall'
.../torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
.../torch/install/share/lua/5.1/nn/Sequential.lua:55: in function 'updateGradInput'
neuralstyle_seg.lua:232: in function 'opfunc'
.../torch/install/share/lua/5.1/optim/lbfgs.lua:66: in function 'lbfgs'
neuralstyle_seg.lua:253: in function 'main'
neuralstyle_seg.lua:546: in main chunk
[C]: in function 'dofile'
.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x55b757411610
WARNING: If you see a stack trace below, it doesn't point to the place where this error occurred. Please use only the one above.
stack traceback:
[C]: in function 'error'
.../torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors'
.../torch/install/share/lua/5.1/nn/Sequential.lua:55: in function 'updateGradInput'
neuralstyle_seg.lua:232: in function 'opfunc'
.../torch/install/share/lua/5.1/optim/lbfgs.lua:66: in function 'lbfgs'
neuralstyle_seg.lua:253: in function 'main'
neuralstyle_seg.lua:546: in main chunk
[C]: in function 'dofile'
.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x55b757411610
As far as I know, I followed the procedure in README, so this seems to be either a bug or there are something undocumented that I should do before running gen_all.py.
martinbenson commented
It could be worth trying with the cudnn flags set if you've not already.
MicaelCarvalho commented
Response to a deleted comment : This project doesn't seem to be compatible with LUA 5.2 (luaL_openlib issues).
@martinbenson : could you be more precise?
Thanks.
martinbenson commented
The project readme mentions two cudnn flags you can add to improve performance and memory usage
themightyoarfish commented