keras-team/keras-applications

ValueError: Unable to create group (name already exists)` when saving MobilenetV3 model to .h5

robert-hardwick opened this issue · 2 comments

robert-hardwick commented

Summary

Created an instance of MobileNetV3Small, attempted to save model with weights to .h5 file and after loading back again the model gives a different output or with tensorflow.keras fails with error.

Issue not present with save_format="tf" / SavedModel

In tensorflow.keras this gave error of:
ValueError: Unable to create group (Name already exists)

After reading about this error it leads me to believe there is an issue with h5 file format and layers with the same name. I tried the same code but with mobilenet_v2 and there is no problem, so my conclusion is that the mobilenetv3 implementation is using duplicated layer names somewhere.

Environment

  • Python version: Python 3.8.2
  • Keras version:
  • Keras-applications version: HEAD of git repo ( bc89834 )
  • Keras backend with version:

Logs or source codes for reproduction

Code :

   from keras.applications import mobilenet_v3

    img_data = process_image("peacock.jpg")
   
    #build mobilenet model and predict test image
    mobilenet_model = mobilenet_v3.MobileNetV3Small(input_shape=(224, 224, 3))
    predictions = mobilenet_model.predict(img_data, batch_size=1)
    print(parse_predictions(predictions))

    # save model to h5
    mobilenet_model.save(cmd_args.out_saved_model)
    
    # load saved file and do perform prediction
    loaded_model = load_model(cmd_args.out_saved_model, custom_objects={"hard_swish": hard_swish})
    predictions = loaded_model.predict(img_data, batch_size=1)
    print(parse_predictions(predictions))

When using tensorflow.keras

{'peacock\n': 0.98741907, 'suspension bridge\n': 0.0005626393, 'punching bag\n': 0.00051906594}
Traceback (most recent call last):
  File "concatenate_networks.py", line 102, in <module>
    main()
  File "concatenate_networks.py", line 63, in main
    mobilenet_model.save("test.h5", save_format="h5")
  File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/engine/network.py", line 1007, in save
    save.save_model(self, filepath, overwrite, include_optimizer, save_format,
  File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/save.py", line 111, in save_model
    hdf5_format.save_model_to_hdf5(
  File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
    save_weights_to_hdf5_group(model_weights_group, model_layers)
  File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 625, in save_weights_to_hdf5_group
    g = f.create_group(layer.name)
  File "/home/robert/tf2/lib/python3.8/site-packages/h5py/_hl/group.py", line 68, in create_group
    gid = h5g.create(self.id, name, lcpl=lcpl, gcpl=gcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5g.pyx", line 161, in h5py.h5g.create
ValueError: Unable to create group (name already exists)

When using keras ( It fails silently and simply outputs incorrect values )

{'peacock\n': 0.98741907, 'suspension bridge\n': 0.0005626393, 'punching bag\n': 0.00051906594}
{'peacock\n': 0.917123, 'little blue heron\n': 0.008386477, 'siamang\n': 0.0042926436}

Notice 0.98741907 != 0.917123

robert-hardwick commented

Have debugged further

input_1
Conv_pad
Conv
Conv/BatchNorm
activation
expanded_conv/depthwise/pad
expanded_conv/depthwise
Traceback (most recent call last):
  File "concatenate_networks.py", line 97, in <module>
    main()
  File "concatenate_networks.py", line 58, in main
    mobilenet_model.save(cmd_args.out_saved_model)
  File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/engine/network.py", line 1007, in save
    save.save_model(self, filepath, overwrite, include_optimizer, save_format,
  File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/save.py", line 111, in save_model
    hdf5_format.save_model_to_hdf5(
  File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
    save_weights_to_hdf5_group(model_weights_group, model_layers)
  File "/home/robert/tf2/lib/python3.8/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 626, in save_weights_to_hdf5_group
    g = f.create_group(layer.name)
  File "/home/robert/tf2/lib/python3.8/site-packages/h5py/_hl/group.py", line 68, in create_group
    gid = h5g.create(self.id, name, lcpl=lcpl, gcpl=gcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5g.pyx", line 161, in h5py.h5g.create
ValueError: Unable to create group (name already exists)

Appears that it doesn't like the layer name "depthwise/pad" before the parent group "depthwise"

keras-applications/keras_applications/mobilenet_v3.py

Lines 161 to 162 in bc89834

x = layers.ZeroPadding2D(padding=correct_pad(backend, x, kernel_size),
name=prefix + 'depthwise/pad')(x)

Guess that's something to do the h5 heirarchical data structure.

Renaming depthwise layer here

keras-applications/keras_applications/mobilenet_v3.py

Lines 163 to 167 in bc89834

x = layers.DepthwiseConv2D(kernel_size,
strides=stride,
padding='same' if stride == 1 else 'valid',
use_bias=False,
name=prefix + 'depthwise')(x)
to "depthwise/DepthwiseConv" fixes the error mentioned above, however the predictions are still different compared to saving as SavedModel.

Might be 2 separate issues here.

robert-hardwick commented

Again, the above debug was using tensorflow keras, had assumed that keras was silently failing but since fixing the layer naming the problem still persists with same symptoms as kera. Will use keras for further debugging.