Feature request: add local and reformer
samvanstroud opened this issue · 1 comments
samvanstroud commented
Thanks for this repo. Is there a possibility of adding your existing local attention and reformer implementations here?
I'm hoping they may also be able to be updated to take advantage of the upcoming attention mask support for the meff kernel in PT2.1.
lucidrains commented
yeah, I do have plans to make it so one can register custom transformer blocks. probably will be tested with mixture of experts first https://github.com/lucidrains/st-moe-pytorch, but will prob also consider local attention