Implementing grad-cam in models that use dropout

[thiippal]

Hey,

I'm attempting to implement grad-cam into my project, which uses a different architecture than VGG16. The model, which has just two convolutional blocks (see here under MiniVGGNet), uses dropout to combat overfitting. Note that this model still uses Theano; I'm moving to TensorFlow now.

Now, if I've understood correctly, Keras must be told whether to operate in training (include dropout) or testing (exclude dropout) mode, as Francois Chollet describes here.

I've therefore included K.learning_phase (which has been set to zero, that is, testing mode) into the gradient function in grad-cam:

gradient_function = K.function([model.layers[0].input, K.learning_phase()], [conv_output, grads])
output, grads_val = gradient_function([image, K.learning_phase()])

This, however, returns the following error:

Traceback (most recent call last):
File "grad-cam-debug.py", line 90, in <module>
cam  = grad_cam(model, preprocessed_input, predicted_class, "maxpooling2d_2")
File "grad-cam-debug.py", line 58, in grad_cam
output, grads_val = gradient_function([image, K.learning_phase()])
File "/Users/tuomo/.virtualenvs/keras-tf/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1040, in __call__
updated = session.run(self.outputs + [self.updates_op], feed_dict=feed_dict)
File "/Users/tuomo/.virtualenvs/keras-tf/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 717, in run
run_metadata_ptr)
File "/Users/tuomo/.virtualenvs/keras-tf/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 872, in _run
+ e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a int into a Tensor.

Any idea what might be the issue here?

[jacobgil]

When calling gradient_function, the actual learning phase should be a scalar.
Please try changing the line
output, grads_val = gradient_function([image, K.learning_phase()])
To:
output, grads_val = gradient_function([image, 0])
Does that work?

[thiippal]

Hi,

output, grads_val = gradient_function([image, 0])

returns the same error message:

Traceback (most recent call last):
File "grad-cam-debug.py", line 90, in <module>
cam  = grad_cam(model, preprocessed_input, predicted_class, "maxpooling2d_2")
File "grad-cam-debug.py", line 58, in grad_cam
output, grads_val = gradient_function([image, 0])
File "/Users/tuomo/.virtualenvs/keras-tf/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1040, in __call__
updated = session.run(self.outputs + [self.updates_op], feed_dict=feed_dict)
File "/Users/tuomo/.virtualenvs/keras-tf/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 717, in run
run_metadata_ptr)
File "/Users/tuomo/.virtualenvs/keras-tf/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 872, in _run
+ e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a int into a Tensor.

I guess the full code would be more helpful here; I commit it to my repo (thanks for the suggestion by the way!) tomorrow.

[jf003320018]

@thiippal You should check your code carefully. I have run this code, and it works well.

[thiippal]

@jf003320018 Which code are you referring to?

@jacobgil I've added the full code for grad-cam on this branch.

[jacobgil]

I took a look at the branch above (only ran with dummy weights since there were no tensorflow format weights).

The problem seems to be the K.set_learning_phase(0) line. Drop that line and it should be ok.

Also in the branch you shared there are a few dimension problems, it seems that MiniVGGNet was called with wrong parameters- (150, 150, 3, 3) instead of (3, 150, 150, 3).
Also replace instances of 224 with 150.

[jf003320018]

Traditionally, dropout do not used for convolutional and pooling layers. So it is not needed to use K.set_learning_phase in the code.

[thiippal]

Hi @jacobgil, unfortunately the cam branch was behind a few commits, so the TensorFlow weights were missing. I also did not get far enough to resize the images from the canonical ImageNet size, but thanks for pointing that out!

Removing K.set_learning_phase(0) helped, and the script now works with my model and weights.

The output, however, is quite interesting: most images produce a class activation map with red colour only, such as the one below:

Another image, in turn, seems to be working well, as passing different values to the category_index variable produces different kinds of activation maps.

My up-to-date code may be found here ... thanks for your support & patience!

[jacobgil]

Hi @thiippal,
The issue seems to be the preprocessing of the image.
Looks like your model was trained on pixels in the range [0, 1], but the images fed were in the range [0, 255].
Change load_image to something like this, and multiply by 255 when used for visualizing:

def load_image(path):

original = keras.preprocessing.image.load_img(path, target_size=(150, 150))  # Load image and resize to 150x150

array = img_to_array(original, dim_ordering='tf')  # Convert image to numpy array

normalized = array.astype("float") / 255.0  # Normalize the array into range 0...1

reshaped = np.expand_dims(normalized, axis=0)  # Reshape for input for CNN    

return reshaped

An example output i'm getting looks like this:

Btw what are the 3 categories in the output?

[thiippal]

Hi Jacob,

thanks – should have figured that out: I faced & solved the same issue when originally using your CAM implementation. Now the script returns completely (well perhaps always not so) sensible results. :-)

The three classes correspond to T-72 tanks, BMP APCs and 'other', i.e. street and countryside scenes without none of the aforementioned classes.

I think this issue can be closed now. Hope this will help someone else implementing Grad-CAM ... thanks for your support!

[absentm]

Thanks a lot, @jacobgil . you answer:

When calling gradient_function, the actual learning phase should be a scalar.
Please try changing the line
output, grads_val = gradient_function([image, K.learning_phase()])
To:
output, grads_val = gradient_function([image, 0])
Does that work?

is ok, and helped me in my code:

iterate = K.function([input_img, K.learning_phase()], [loss, grads])
loss_value, grads_value = iterate([input_img_data, 1])