A NeurIPS paper on efficient architecture
renll opened this issue · 1 comments
renll commented
Thanks for the great survey! Could you please include a discussion of this work from Microsoft and UIUC? It proposes a general modular activation mechanism, SMA, that unifies previous works on MoE, adaptive computation, dynamic routing and sparse attention, and further applies SMA to develop a novel architecture, SeqBoat, to achieve SoTA quality-efficiency trade-off on Long Range Arena.
SUSTechBruce commented
Thanks for the suggestion and congrats on your NeurIPS paper!
We've added it to the github paperlist and will update it to survey in next version~