Load_model memory leak

[enricopal]

Hello,
I am running a server application that uses Keras and I have noticed that the memory allocated to the process is increasing with the number of calls to the API, significantly increasing the response time. I am using Ubuntu 16.04, python 3.6.0, Tensorflow as a backend.
I have created a small experiment to measure the size of the allocated memory when loading a model multiple times (I am using Ubuntu 16.04, python 3.6.0, Tensorflow as a backend):

from keras.models import load_model
import time
from pympler import asizeof
model_file = 'model.h5'

for i in range(10):
model = load_model(model_file)
print(asizeof.asizeof(model))
del model
time.sleep(2)

The output is:

8847736
17060064
25370520
33237832
41498512
49956848
58217816
66084752
74674104
82541328

You can get the model file here to reproduce the experiment: https://drive.google.com/file/d/0B0f321vV55z9T3gzV0ZLWTRBa3c/view?usp=sharing.
It seems that, even deleting the model variable, not all the memory is deallocated.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[yanpanlau]

mind if I ask any update on this? I am seeing the same problem.

[enricopal]

The workaround we had found using Flask was to load all the models before the first request, using the decorator:
@app.before_first_request
def load_model():
I can suggest you to look into that.

[VincentPisztora]

Hi there - is there any other solution besides loading all models at the start?

I am re-loading weights from the previous epoch when I hit a batch that gives a nan after the train_on_batch step and am running into the same issue. So I can't use the workaround suggested by enricopal because the weights that need loading aren't known before I start the training.

Thanks!

[tatevmejunts]

Hello everyone.

I am running Flask application and using Keras load_model() function to load several models. I also need to use tf.Graph().as_default() so K.clear_session() doesn't help me. After one week memory fills up completely. Does anyone find out how to prevent memory leak?

Thank you in advance

[rodrigoruiz]

Same issue here with the latest version (2.3.1), but running on jupyter notebook. No fix yet? It's been 2 years...

[leenremm]

Same issue here. An update on this would be much appreciated.