error occurred when run tutorial on custom model

[ascdqz]

Hi
I'm trying to use zennit to draw a lrp heatmap for my self trained vgg11 model. I followed the tutorial on the document, but when I use my models(which just trained the last layer and changed output dim to 6) will have errors. It will say that input type and weight type should be the same. Then I changed my data to cuda mode, the second error "tuple has no attribute detach" occurred. Is this tutorial only for pure pretrained model? Can you tell me how to implement it on my slightly customed vgg11 model?
Thank you!

[chr5tphr]

Hey @ascdqz

could you maybe share the code so I can have a better look at it?
Otherwise, from what I can see, make sure to move your model weights to the same device as the input.
Also, does your model output a tuple? This is currently not supported with attributors, so you can either make sure it is not a tuple, or directly use the composite.

[ascdqz]

Hi @chr5tphr
Thank you for your response. You were right, my model output is a tuple. Do you think it's normal to have a tuple as a vgg model output? I think in my case(listed below), the second tensor in the tuple doesn't seem very necessary(all zeros), so I added [0] in the attribution.py(listed below) to see any changes. And it gave me another error as I listed below. I already checked that my model and data are both in gpu mode and colab is also in gpu mode. And I looked at the documents, and there are codes as below seems using composite only. I tried using this, but it also gives an error that "input should be a tensor, not a tuple". Can you tell me how to use composite only and bypass attributors? I can't find these on the document. I'm not very experienced with this. Thank you for your time! And here is my code, the zennit part is at the bottom. https://colab.research.google.com/drive/1BNkPw908xzv3ph57_2Po754sCxhL9Wye?usp=sharing

[ascdqz]

And the loss function is in cuda too

[chr5tphr]

Hey @ascdqz

I checked out your code and could not reproduce your "input should be a tensor, not a tuple" error.

I was referring to the second approach using .context(). I tried it out on your model:

Code

import torch
import torch.nn as nn

from zennit.attribution import Gradient
from zennit.composites import EpsilonGammaBox
from zennit.torchvision import VGGCanonizer


class VGG(nn.Module):
    def __init__(self, features, output_dim):
        super().__init__()

        self.features = features

        self.avgpool = nn.AdaptiveAvgPool2d(7)

        self.classifier = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(4096, output_dim),
        )

    def forward(self, x):
        x = self.features(x)
        x = self.avgpool(x)
        h = x.view(x.shape[0], -1)
        x = self.classifier(h)
        return x, h


vgg11_config = [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M']

vgg13_config = [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512,
                512, 'M']

vgg16_config = [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512,
                'M', 512, 512, 512, 'M']

vgg19_config = [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512,
                512, 512, 'M', 512, 512, 512, 512, 'M']


def get_vgg_layers(config, batch_norm):

    layers = []
    in_channels = 3

    for c in config:
        assert c == 'M' or isinstance(c, int)
        if c == 'M':
            layers += [nn.MaxPool2d(kernel_size=2)]
        else:
            conv2d = nn.Conv2d(in_channels, c, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(c), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = c

    return nn.Sequential(*layers)


def main():
    SEED = 1234
    OUTPUT_DIM = 6

    torch.manual_seed(SEED)

    vgg11_layers = get_vgg_layers(vgg11_config, batch_norm=True)
    model = VGG(vgg11_layers, OUTPUT_DIM)
    model.eval()

    data = torch.randn(18, 3, 224, 224)

    output = model(data)

    canonizers = [VGGCanonizer()]
    composite = EpsilonGammaBox(low=-3., high=3., canonizers=canonizers)
    target = torch.eye(6)[[0] * data.shape[0]]

    with composite.context(model) as modified:
        data.requires_grad = True
        output, features = modified(data)
        attribution, = torch.autograd.grad(output, data, target)


if __name__ == '__main__':
    main()

This worked for me, although I did not use cuda.
Does it work for you if you implement it using composite.context() like this?

[ascdqz]

Hi @chr5tphr
Thank you for your suggestion. It can finally run through all the codes! But I noticed that the printed prediction is always zero(I tried changing class index). The model and input and the model output all seem functional. And my test loss and test accuracy all seem pretty decent. Can you think of any possible reason for that? And my code is still the same:
https://colab.research.google.com/drive/1BNkPw908xzv3ph57_2Po754sCxhL9Wye?usp=sharing

[chr5tphr]

Did you try out different images with different ground truth classes?
If you try it out for all images, I expect you will get the same accuracy.

Since I do not know your dataset, there's also the possibility that your dataset is heavily unbalanced (i.e. 74.07% are of label 0) and thus your model simply always predicts class 0. Though, unless the prediction is in some way erroneously changed by the composite (which I do not think is the case here) this is out-of-scope for Zennit.

Also note, before you forget, to set the correct labels for your target variable, which is the class you are trying to attribute.

[ascdqz]

I think I misunderstood something at the beginning. It works very well now.
I think this problem is solved. Thank you for your help.