Question with logit_scale on your code

[HuisuIntwnr]

Thank you for your github.
I appreciate your paper and code sharing on github!

I have a question with logit_scale on your code.
There is a logit_scale in 'models/openad_pn2.py' like this 'self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))' on class OpenAD_PN2.

I wonder how this self.logit_scale was determined. Can you explain it in detail?

Thank you for your Github!

[toannguyen1904]

Hi @HuisuIntwnr. Thank you very much for your interest in our work.

For the conciseness, we did not mention the logit_scale in our paper. In our implementation, the logit_scale is a learnable temperature parameter, which is updated during training. The purpose of this parameter is to control the range of the logits in the softmax. The use of the logit_scale is mentioned in the CLIP paper, which you can refer to here. If you wish to delve deeper into the importance of this temperature parameter, you can refer to this paper as well.

Best regards,
Toan.