berniwal/swin-transformer-pytorch

A question about qk_scale

Closed this issue · 0 comments

Sample-design-alt commented

Hello @berniwal ,
I have a question about this:

swin-transformer-pytorch/swin_transformer_pytorch/swin_transformer.py

Line 76 in c921ebf

self.scale = head_dim ** -0.5

what's the function of the scale?I can't understand why do this.

Best regards