lucidrains/x-transformers

encoder.to_logits.weight doesn't update

Closed this issue · 3 comments

guillaumeguy commented

It seems that parameter encoder.to_logits.weight in an Encoder/Decoder is not part of the forward pass. This creates problems when running DDP.

Should these weights be removed in such case?

Reproducible example:

lucidrains commented

@guillaumeguy ah yes, that shouldn't be there

could you try 1.35.2?

guillaumeguy commented

Works great! Thank you for the quick turnaround!

  • 👍1
lucidrains commented

@guillaumeguy you are one of the few using a full encoder decoder transformer

how are you using it?