graykode/nlp-tutorial

Why is first parameter `src_vocab_size`?

yangdechuan opened this issue · 1 comments

yangdechuan commented

nlp-tutorial/5-1.Transformer/Transformer(Greedy_decoder)-Torch.py

Line 141 in 3b3a80d

self.pos_emb = nn.Embedding.from_pretrained(get_sinusoid_encoding_table(src_vocab_size, d_model),freeze=True)

position encoding table should be (src_len, d_model). Why (src_vocab_size, d_model) here?

graykode commented

Oops.. Thanks for right catching
In Encoder src_len+1 is right, and as same tgt_len+1 in Decoder.
I will fixed as soon as possible Thanks