Resource Exhausted Error while testing
prathmeshrmadhu opened this issue · 0 comments
prathmeshrmadhu commented
ResourceExhaustedError` (see above for traceback): OOM when allocating tensor with shape[1,64,1600,1330] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node Conv2D_10 (defined at /localhome/prathmeshmadhu/work/EFI/Code/arbitrary_style_transfer/encoder.py:95) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_10-0-TransposeNHWCToNCHW-LayoutOptimizer, encoder/conv1_2/kernel/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node clip_by_value/_77}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_487_clip_by_value", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
I am trying to generate some 40000+ different stylized images and it generates some 220+ images and then always throws this resources error. My hardware details are as follows :
GPU : GeForce GTX 970, 4GB.
CPU : 8 cores, 64 GB.
Do let me know if you need any further details.
Thanks.