Calling get_activations in a loop causing problems

[hbarovertwo]

Hopefully someone can point me in the right direction.
I'm using get_activations to do some computations on different layer's output and do something with the result. The workflow is like:

model = VGG16()
model.compile(...)
for item in list:
load_img
get_activations(model, img)
# do stuff with activation layers

each time I loop through takes longer and longer. I've narrowed it down to the get_activations call that is causing the issue. It takes more time each iteration to get the activations. What is a workaround here?

[hbarovertwo]

@philipperemy I've narrowed it down to the _evaluate call in get_activations.

[philipperemy]

@hbarovertwo Could it be possible that the model gets re-compiled everytime the function is called? Do you have a code snippet that I can try to investigate too?

[philipperemy]

300 calls. y-axis is seconds.

I don't see any major problem here. It does not look like it's increasing over time. There are some outliers though.

from keras.applications.vgg16 import VGG16
from keras.applications.vgg16 import decode_predictions
from keras.applications.vgg16 import preprocess_input
from keras.preprocessing.image import img_to_array

model = VGG16()

from PIL import Image
import requests
from io import BytesIO

url = 'https://upload.wikimedia.org/wikipedia/commons/thumb/1/14/Gatto_europeo4.jpg/250px-Gatto_europeo4.jpg'
response = requests.get(url)
image = Image.open(BytesIO(response.content))
image = image.crop((0, 0, 224, 224))
image = img_to_array(image)
image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))
image = preprocess_input(image)
yhat = model.predict(image)
label = decode_predictions(yhat)
label = label[0][0]
print('{} ({})'.format(label[1], label[2] * 100))

import keract

model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

from time import time
import matplotlib.pyplot as plt
times = []
for i in range(1000):
    a = time()
    activations = keract.get_activations(model, image)
    times.append(time() - a)
    print(times[-1], 'seconds')
    if i % 100 == 0 and i > 0:
        plt.plot(times)
        plt.show()

[hbarovertwo]

I'll do a run now and plot the times as well. I compile the model once outside the loop as you do, and loop through a list of unique images. It seems the bug appears when getting activations for different images.

[hbarovertwo]

And here is the snippet of code that causes the behavior @philipperemy

import os
import numpy as np
from keract import get_activations
from keras.applications.vgg16 import VGG16
from keras.applications.vgg16 import preprocess_input
from keras.preprocessing.image import load_img, img_to_array
from tqdm import tqdm
from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

model = VGG16(weights='imagenet')
model.compile(optimizer='adam', loss='categorical_crossentropy')

def preprocessing(input_image):
    img = load_img(input_image, target_size=(224,224))
    img = img_to_array(img)
    img = np.expand_dims(img, axis=0)
    img = preprocess_input(img)
    return img

db = '/home/rahul/new_vienna/'
with open('/home/rahul/similarity/trainset.txt','r') as f:
    im_files = eval(f.read())
files = [db+i for i in im_files]

from time import time
import matplotlib.pyplot as plt
times = []

for i in range(len(files)):
    a = time()
    im = preprocessing(files[i])
    activations = get_activations(model, im)
    times.append(time()-a)
    print(times[-1], 'seconds')
    if i % 100 == 0 and i > 0:
        plt.plot(times)
        plt.savefig('foo.png')
        print('figure saved')

[hbarovertwo]

And it quickly becomes problematic. Oddly enough I get the same behavior using only 1 image and calling get_activations on it in the same loop as you. When I pip installed keract it also forced an install of a different keras version, the related package versions I have are tensorflow-gpu 1.13.1, keras 2.3.1, keract 2.8.4

[philipperemy]

@hbarovertwo that's weird. Could it be because of the preprocessing or the fact that the model gets compiled every single step?

[philipperemy]

Can you try with an update version of keras? I can't reproduce the problem myself... And can you provide a ZIP of your images or something that can help me reproduce the error?

[hbarovertwo]

@philipperemy The problem seemed to be in the activations call, I tested the times for each block in the loop and that was the only block that grew with iterations. I was only interested in the output of 3 intermediate layers so I ultimately just used a backend function to run inputs through and catch the outputs of interest. Using the same package versions as mentioned above, going this route didn't produce a similar effect and I was able to run the loop in more or less constant time.

Closing this issue for now as I found a work around and I've moved on to another project. Thanks for the help.

[philipperemy]

@hbarovertwo Ok thanks for your analysis. If others people encounter this problem, we will re-open this issue. Thanks again!