Question: masking in token shifting
Opened this issue · 1 comments
pfeatherstone commented
In token shifting, you explicitly zero out masked items:
x-transformers/x_transformers/x_transformers.py
Lines 554 to 555 in 5ce82c9
Is this strictly necessary? Since we are shifting right, the shifted tokens should be valid right?
Or is this accounting for items masked on the left? In which case you might be shifting and adding with an invalid token?
I noticed that RecurrentMemoryTransformer
didn't do this:
Hence why I'm asking if it's strictly necessary.
lucidrains commented
@pfeatherstone i think i allow for bidirectional shifting, maybe that's why
i can check later