SamLynnEvans/Transformer

Did not multiply embedding weights by sqrt(d_model)

orena1 opened this issue · 4 comments

orena1 commented

Hi,
In this line:

Transformer/Embed.py

Line 12 in 37bf492

return self.embed(x)

I think you need to multiply the embedding by sqrt(d_model)
image

fabrahman commented

@orena1 Hi, the implementation also didn't share the embedding weights, right?

fabrahman commented

@orena1 The code actually has * math.sqrt(self.d_model) in the positional embedding class. In forward method.

  • 👍1
  • 😄1
zhangxixi0904 commented

Did somebody know the reason for multiplying embedding weights by sqrt(d_model)？

  • 👍1
wangzelin-em commented

@orena1 Hi, the implementation also didn't share the embedding weights, right?

Yes, the implementation didn't share the embedding weights.