nik-dim/tall_masks

DTD test set performance

kasurashan opened this issue · 3 comments

kasurashan commented

Hello.

Thank you for sharing this interesting research and the codes.

https://github.com/nik-dim/tall_masks/blob/db2488ad4a4f1151b096ad72019a6582db1f60fc/results/single_task/ViT-B-32/nonlinear_ft_accuracies.json

image

It seems like the individual model's performance on the DTD test set is unexpectedly high. I'm curious as to why these are coming out this way. Could you explain the possible reasons for this?

Thank you.

wang-kee commented

Hi,

Thanks for your interest in our work.

We have noticed that there was a issue in generating the split of DTD test set. We refer to mlfoundations/task_vectors#1 for generating it.

We will update our checkpoints soon with the updated model checkpoint and masks.

Please feel free to let us know if there are anything else we could help you with.

  • 👍1