TempleX98/MoVA

Dinov2 seems to fail in some cases

liuyifan22 opened this issue · 2 comments

liuyifan22 commented

Hello! Thanks for your marvellous work. I'm learning from your eval code, but have been confronted by some unxepected errors. It seems that Dinov2 is not functioning properly, with the ERROR info:

mova/mova/model/vision_experts/dinov2/modeling_dinov2.py", line 415, in forward
self.norm1(hidden_states), # in Dinov2, layernorm is applied before self-attention
xxx
xxx
RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [1536] and normalized_shape = [1024]

Here is the config for my downloaded dinov2:
image

I guess there maybe some incompatibility between my downloaded Dinov2. Would you please provide your dinov2 model&checkpoint to help me out? Thanks!

TempleX98 commented

I am sorry for the mistake. The config.json file has an incorrect model name of DINOv2. We should be using facebook/dinov2-giant instead of the large model. I’ve fixed the issue now.

  • ❤️1
liuyifan22 commented

Thanks for your timely reply! After taking on the giant version, the eval runs smoothly.

  • 👍1