Change the network

[Xlgd]

Hi, if I want to use the BN-Inception instead of ResNet50, what else do I need to change besides adding the BN-Inception structure in netlib.py and modifying the model definition in main.py? Thank you.

my conda env:

• Python 3.7.9
• PyTorch 1.4.0 and Cuda 10.2
• Faiss-gpu 1.6.3
• Scikit Learn 0.23.2
• Scipy 1.5.2

[Xlgd]

English is not my native language; please excuse typing errors.

My goal is to use BN-Inception network to get the respective Recall@K for CUB, Cars and InShop datasets.

I replace ResNet50 with BN-Inception and allocate 25% of the training set for the validation set, I use Result_Runs.sh to train the network on the training set and verify the effect with the validation set. After the training I then run it on the test set using the checkpoint to get the final Recall@K.

Here's how I modified the code to partition the Cub and Cars datasets (in datasets.py):

train, val, test = keys[:(len(keys)//8) * 3], keys[(len(keys)//8) * 3:len(keys)//2], keys[len(keys)//2:]

the origin code is:

train,test = keys[:len(keys)//2], keys[len(keys)//2:]

For the InShop dataset, I selected one part of the training set as query, another part as gallery, both together as a validation set, and the rest as a training set.
I only modified the code in netlib.py, datasets.py and main.py.

In the end, the Recall@K is very low and weird. I've read README.md carefully, my data dir structure matches the requirements, my conda environment is included in the last comment, and I've tried my best to find the answer on the web, but I haven't been able to solve the problem, and I don't know where I went wrong.

My results are as follows:

CUB
Validation:

NMI: 0.201 
F1: 0.109
Recall @ 1: 0.223
Recall @ 2: 0.330
Recall @ 4: 0.455
Recall @ 8: 0.616
Recall @ 16: 0.787
Recall @ 32: 0.923

Test:

NMI: 0.217
F1: 0.0215
Recall @ 1: 0.056
Recall @ 2: 0.089
Recall @ 4: 0.143
Recall @ 8: 0.228
Recall @ 16: 0.347
Recall @ 32: 0.510

Cars
Validation:

NMI: 0.261
F1: 0.127
Recall @ 1: 0.384
Recall @ 2: 0.515
Recall @ 4: 0.642
Recall @ 8: 0.779
Recall @ 16: 0.894
Recall @ 32: 0.966

Test:

NMI: 0.299,
F1: 0.052
Recall @ 1: 0.240
Recall @ 2: 0.337
Recall @ 4: 0.452
Recall @ 8: 0.583
Recall @ 16: 0.718
Recall @ 32: 0.836

InShop
Validation:

NMI: 0.805
F1: 0.172
Recall @ 1: 0.002
Recall @ 10: 0.012
Recall @ 20: 0.019
Recall @ 30: 0.030
Recall @ 40: 0.037
Recall @ 50: 0.047

Test:

NMI: 0.748
F1: 0.023
Recall @ 1: 0.149
Recall @ 10: 0.288
Recall @ 20: 0.340
Recall @ 30: 0.377
Recall @ 40: 0.404
Recall @ 50: 0.425

Afterwards, in order to find out the cause of the problem, I switched the network to ResNet50 and made sure that all other configurations remained the same (training set, validation set, test set, Result_Runs.sh, and the environment), and the final data was as follows:

CUB
Validation:

NMI: 0.809
F1: 0.668
Recall @ 1: 0.883
Recall @ 2: 0.932
Recall @ 4: 0.962
Recall @ 8: 0.982
Recall @ 16: 0.990
Recall @ 32: 0.995

Test:

NMI: 0.653
F1: 0.343
Recall @ 1: 0.599
Recall @ 2: 0.718
Recall @ 4: 0.820
Recall @ 8: 0.891
Recall @ 16: 0.938
Recall @ 32: 0.967

Cars
Validation:

NMI: 0.651
F1: 0.446
Recall @ 1: 0.803
Recall @ 2: 0.885
Recall @ 4: 0.937
Recall @ 8: 0.971
Recall @ 16: 0.986
Recall @ 32: 0.996

Test:

NMI: 0.587
F1: 0.262
Recall @ 1: 0.666
Recall @ 2: 0.782
Recall @ 4: 0.866
Recall @ 8: 0.922
Recall @ 16: 0.953
Recall @ 32: 0.977

InShop
Validation:

NMI: 0.850
F1: 0.280
Recall @ 1: 0.001
Recall @ 10: 0.008
Recall @ 20: 0.016
Recall @ 30: 0.023
Recall @ 40: 0.033
Recall @ 50: 0.041

Test:

NMI: 0.843
F1: 0.183
Recall @ 1: 0.729
Recall @ 10: 0.914
Recall @ 20: 0.940
Recall @ 30: 0.952
Recall @ 40: 0.959
Recall @ 50: 0.965

I hope someone can help me figure out the cause of the problem so that I can get the correct recall for all three datasets, thank you for your help!

[Confusezius]

Hi there! Sorry for not replying earlier!

It's hard to diagnose the issue directly, but the following things should be checked:

1. Is the output of the network normalized? I.e. the ResNet50 uses normalized embeddings, and distance-based sampling requires those.
2. The Inception-BN should use a default embedding dimension of 512.
3. Have you considered freezing the Batchnorm layers? A majority of literature implementations do this as well.

The libraries used shouldn't be the problem, it's likely an issue with the setup, as the ResNet-Setting seems to be working :).

[Xlgd]

Thank you very much for your response!
I will check my code tomorrow and respond to you.

[Xlgd]

Thank you for your reply again.
I just checked my code.

1. the output of the network has been normalized, just like ResNet50.
   In netlib.py, NetworkSuperClass_BNInception class:

def forward(self, x, is_init_cluster_generation=False):
        itermasks, out_coll = [],{}

        #Compute First Layer Output
        x = self.model.pool1_3x3_s2(self.model.conv1_relu_7x7(self.model.conv1_7x7_s2_bn(self.model.conv1_7x7_s2(x))))

        if is_init_cluster_generation:
            #If the first clusters before standardization are computed: We use the initial layers with strong
            #average pooling. Using these, we saw much better initial grouping then when using layer combinations or
            #only the last layer.
            conv2_3x3_reduce_out = self.model.conv2_3x3_reduce(x)
            conv2_3x3_reduce_bn_out = self.model.conv2_3x3_reduce_bn(conv2_3x3_reduce_out)
            conv2_relu_3x3_reduce_out = self.model.conv2_relu_3x3_reduce(conv2_3x3_reduce_bn_out)
            conv2_3x3_out = self.model.conv2_3x3(conv2_3x3_reduce_bn_out)
            conv2_3x3_bn_out = self.model.conv2_3x3_bn(conv2_3x3_out)
            conv2_relu_3x3_out = self.model.conv2_relu_3x3(conv2_3x3_bn_out)
            x = self.model.pool2_3x3_s2(conv2_3x3_bn_out)
            x = torch.nn.functional.avg_pool2d(x,18,12)
            x = torch.nn.functional.normalize(x.view(x.size(0),-1))

            return x

And at the last in the forward function:

#Store the final conv. layer output, it might be useful.
            out_coll['Last'] = x
            for out_mode in self.out_modes:
                mod_x = self.model.last_linear[out_mode](x)
                out_coll[out_mode] = torch.nn.functional.normalize(mod_x, dim=-1)
            return out_coll

2. I use the default embedding dimension of 512.
   In all Parameter_Info.txt:

embed_sizes
	[512, 512]

3. I have freezed the Batchnorm layers of BNInception.
   In NetworkSuperClass_BNInception class:

class NetworkSuperClass_BNInception(nn.Module):
    def __init__(self, opt):
        super(NetworkSuperClass_BNInception, self).__init__()

        self.pars = opt

        if not opt.not_pretrained:
            print('Getting pretrained weights...')
            self.model = ptm.__dict__['bninception'](num_classes=1000, pretrained='imagenet')
            print('Done.')
        else:
            print('Not utilizing pretrained weights!')
            self.model = ptm.__dict__['bninception'](num_classes=1000, pretrained=None)


        for module in filter(lambda m: type(m) == nn.BatchNorm2d, self.model.modules()):
            module.eval()
            module.train = lambda _: None


        ### Set Embedding Layer
        in_feat = self.model.last_linear.in_features
        self.out_modes, self.embed_sizes   = opt.tasks, opt.embed_sizes
        self.model.last_linear = nn.ModuleDict({task: torch.nn.Linear(in_feat, self.embed_sizes[i]) for i,task in enumerate(self.out_modes)})

Please help me check if I'm doing it right. Thank you!

[Xlgd]

I found an error in BN-Inception class and fixed it.
I will retrain the model and show you the results later.
Thank you!

[Confusezius]

Ok, that's good! Let's hope that that was the issue. Also last note: the network uses Imagenet pretrained weights, right?

[Xlgd]

Ok, that's good! Let's hope that that was the issue. Also last note: the network uses Imagenet pretrained weights, right?

Yes, the model uses it.

if not opt.not_pretrained:
    print('Getting pretrained weights...')
    self.model = ptm.__dict__['bninception'](num_classes=1000, pretrained='imagenet')
    print('Done.')
else:
    print('Not utilizing pretrained weights!')
    self.model = ptm.__dict__['bninception'](num_classes=1000, pretrained=None)

[Xlgd]

The latest data is as follows:

Reall@K(CUB)

1	2	4	8	16	32
0.0562	0.0898	0.1433	0.2282	0.3472	0.5101

Recall@K(Cars)

1	2	4	8	16	32
0.2373	0.3206	0.4278	0.5412	0.6685	0.8023

Recall@K(InShop)

1	10	20	30	40	50
0.1493	0.2882	0.3403	0.3772	0.4041	0.4253

Although I have fixed a small bug in the network structure, it doesn't seem to be the main one, because the Recall@K is still very low and weird.

Let me summarize the current situation:
The BN-Inception network I used has freezed the BatchNorm layers and normalized the output. The embedding size is the default 512, using the pretrained weights.
I used the same training set, validation set, and test set, changed the network to ResNet50, and got the following Recall@K:

Reall@K(CUB)

1	2	4	8	16	32
0.5992	0.7180	0.8207	0.8916	0.9383	0.9677

Reall@K(Cars)

1	2	4	8	16	32
0.6667	0.7826	0.8667	0.9229	0.9535	0.9775

Reall@K(InShop)

1	10	20	30	40	50
0.7294	0.9148	0.9404	0.9527	0.9595	0.9651

I think BN-Inception's Recall@K is strange, but I haven't found the cause of the problem yet.
If I need to change the network structure, is there anything else I should be concerned about?

[Xlgd]

Do you think it's because there are too few training epochs, since I switched the network to BN-Inception, but am still using Result_Runs.sh that you wrote for ResNet50?

[Confusezius]

Hmm, that's weird. While I don't have time rn to check it myself, you should try adjusting two hyperparameters:
adv_weights (e.g. in the range 100, 5000) and random_cluster_pick (e.g. in the range 0.1, 0.5) to see if it's the pseudolabel training that's causing the issue.

Also, try it with an embedding dim of 128 to make the BN-Inception as similar to the resnet one - if that is training, then it means that the higher dimensionality requires some minor changes in the clustering process.

Hope that helps!

[Xlgd]

Thank you for your reply!
I will try it.

[Xlgd]

I increased the epoch and retrained the network. I found that Recall@K is normal and not weird now.
I'm sorry to bother you during this time, thank you very much for your patient answer, thanks!

[Xlgd]

To add a summary, the cause of the problem lies in the following two points.

1. The model is underfitting. Because I changed the network, so I need to adjust the training parameters to achieve better results.
2. InShop validation set of the wrong label, resulting in a very low Recall.

[Confusezius]

Happy to hear it's working now and thanks for the summary! Closing it again, but feel free to reopen if other questions arise :).