gaopengcuhk/Tip-Adapter

How to extend to base-to-novel classes task?

Opened this issue · 4 comments

jingzhengli commented

Hi, This method modifies the parameters of the text encoder, so it cannot extend to base-to-new classes tasks. I would like to know how to address this problem.

ZrrSkywalker commented

Thanks for your interest.
We do not modify the parameters of the original textual encoder. The CLIP's logits calculated by the textual encoder are linearly interpolated with the cache model's.
Can you provide more details about base-to-novel classes task?
Thanks.

jingzhengli commented

你好，我就用中文回复啦。
我也觉得Tip-Adapter不能像有些算法那样可以拓展到新的类别识别的任务。
此外，你在实验中有没有发现一个问题：尽管目前大部分的paper都没有对比完全微调CLIP模型的实验结果，但我发现完全微调基于ResNet的CLIP实验效果非常差，如果是基于ViT的CLIP效果还行。我一直不能理解这个原因是什么。

ZrrSkywalker commented

你好，是这样的，cache model需要新的类别的训练数据才可以构建
完全微调不同backbone的话，性能可能和学习率有关？我们并没有做太多的尝试

caoziyang1997 commented

Can you share the StanfordCars dataset with me? I couldn't find it online.