keras-team/keras-applications

There is a bug using GPU in MobileNetV3Small

asparsa opened this issue · 0 comments

I wanted to change the Implementation of MobilebetV3Small for my use case. so I cloned the keras-applications and tried to run the mobileNetV3 directly from the source. but it had raised errors when I was using GPU
tensorflow Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try to see if a warning log message was printed above.
I tried TPU and CPU and everything was right. so I tried to limit the GPU memory usage manually to solve it as GPU memory was almost full when I got the error. as a result, I had to pick a smaller batch and it didn't work well so I gave up and trained my network with TPU but it took so long.

Summary

there is a bug when you are training your network with GPU

Environment

  • Keras version: 2.9
  • Keras-applications version:1.0.8
  • Keras backend with version:2.9

Logs or source codes for reproduction

model5=models.Sequential()
model5.add(MobileNetV3(stack_fn,1000,input_shape=(480,640,3),model_type='small',alpha=0.75,minimalistic=False,include_top=False,backend=keras.backend))
model5.add(layers.GlobalMaxPool2D())
model5.add(layers.Flatten())
model5.add(layers.Dense(9, activation='tanh'))
model5.add(layers.Dense(3))
model5.build(input_shape=(1,480, 640, 3))