fastai/courses

In lession 1, OOM when using gpu version tensorflow as keras backend, but i think i got enough memory

JasonHanG opened this issue · 1 comments

I Think I got enough gpu memory:
see log "Free memory: 3.55GiB"
for the these part 2.02GiB + 798.06MiB, but it gives me an OOM error.
I'm using tensorflow-gpu==1.3.0 and Keras==2.0.8 on a single GeForce GTX 960 card
This is log:

name: GeForce GTX 960
major: 5 minor: 2 memoryClockRate (GHz) 1.2025
pciBusID 0000:01:00.0
Total memory: 3.94GiB
Free memory: 3.55GiB
2017-10-23 22:07:24.232475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 
2017-10-23 22:07:24.232478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y 
2017-10-23 22:07:24.232483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960, pci bus id: 0000:01:00.0)
Found 23000 images belonging to 2 classes.
Found 2000 images belonging to 2 classes.
Epoch 1/1
2017-10-23 22:07:27.979547: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.02GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-10-23 22:07:38.053482: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 798.06MiB.  Current allocation summary follows.
2017-10-23 22:07:38.053544: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): 	Total Chunks: 1, Chunks in use: 0 256B allocated for chunks. 4B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
....

Use smaller batch_size (32) to solve this.