Model saving issue

[myopengit]

I have trained the model for around 230 epoch on the coco dataset using your "train_coco.py". I saved the model using detr.save_weights('weights.h5'). When testing on the same image, I got different results when I loading back the saved weights... (and the performance is not good at all) It seems this is an issue with keras （keras-team/keras#4875). Can you provide a solution on how to save your model? Thanks.

[PhanTask]

I use save_weights and load_weights to save and load the trained weights and everything works fine for me. I finetuned the model on my own dataset. Did you normalize the input coco images before doing inference? Also you may check if you set training=False for detr when doing inference.

[simpad2409]

Hello, @myopengit . Can I ask you how did you solve it? I think I have the same problem as you. I save and restore the checkpoints with save_weights and load_weights to save and load the trained weights. The checkpoints I retrieve for inference do not give me the same result.

Thanks in advance.

[simpad2409]

@thibo73800 checkpoints don't save real weights. How can I solve this problem? I can't save my model ... Help me. thank you.

[yuh8]

@simpad2409 You probably used custom functions/schedules/layers which cannot be serialized by keras. I have the exact same issue; I cannot use save model, and saving and loading weights give very poor results. The way I got away with this is to used to the following saving procedure.
tf.keras.models.save_model(model, './generator_model/', include_optimizer=False) model_new = tf.keras.models.load_model('./generator_model/', compile=False) model_new.compile(optimizer=get_optimizer(), loss=loss_func_gen) res = model_new.evaluate(data_iterator_test('generator_data/test_data/df_test.csv'), return_dict=True)
Pay attention to "include_optimizer=False". I have used a custom learning rate schedule which is part of the optimizer and cannot be saved using model.save(). By setting this config to false, I am able to save model. In the load_model, I also set the compile to False. This will load the model without optimizer and without compilation. Subsequently, I compiled again and results are the same.

P.S. I am using tensorflow 2.6.0 with keras 2.6.0 on python 3.8.3

[simpad2409]

@yuh8 Thanks a lot for the answer! You have been very kind ... I'll try as soon as possible and I'll let you know.

I'm trying to finetun DETR for people detection, feeding the Crowd Human dataset. The results I get, however, are no better than the original model, albeit verticalized to pedestrian detection. Do you have any advice for me?

I'm using the same code as the "finetune_hardhat.py" file, with batchsize to 32.

Thanks. :)