lucidrains/x-transformers

Allow passing in a pre-existing TokenEmbedding into TransformerWrapper

Waino opened this issue · 0 comments

Allow passing in a pre-existing TokenEmbedding into TransformerWrapper

Currently TransformerWrapper always creates a new TokenEmbedding object.

TransformerWrapper should take an optional keyword argument token_emb.
If not specified or set to the default value None, a new TokenEmbedding is created exactly as before. If a pre-existing TokenEmbedding is specified, it is used as the token_emb.

This feature is useful for parameter sharing, allowing the use of a single shared embedding with multiple Transformers. I have a use case for this feature in modular multilingual machine translation.