linear-attention
There are 17 repositories under linear-attention topic.
BlinkDL/RWKV-LM
RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
happinesslz/LION
[NeurIPS 2024] Official code of ”LION: Linear Group RNN for 3D Object Detection in Point Clouds“
lucidrains/taylor-series-linear-attention
Explorations into the recently proposed Taylor Series Linear Attention
lucidrains/agent-attention-pytorch
Implementation of Agent Attention in Pytorch
lironui/Multi-Attention-Network
The semantic segmentation of remote sensing images
lironui/MAResU-Net
The semantic segmentation of remote sensing images
lucidrains/autoregressive-linear-attention-cuda
CUDA implementation of autoregressive linear attention, with all the latest research findings
glassroom/heinsen_attention
Reference implementation of "Softmax Attention with Constant Cost per Token" (Heinsen, 2024)
gmongaras/Cottention_Transformer
Code for the paper "Cottention: Linear Transformers With Cosine Attention"
BICLab/MetaLA
Offical implementation of "MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map" (NeurIPS2024)
robflynnyh/hydra-linear-attention
Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)
OSU-STARLAB/LeaPformer
[ICML 2024] Official implementation of "LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions."
RWKV-Wiki/rwkv-wiki.github.io
RWKV Wiki website (archived, please visit official wiki)
hp-l33/flash-bidirectional-linear-attention
Triton implement of bi-directional (non-causal) linear attention
gmlwns2000/sea-attention
Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)
mtanghu/LEAP
LEAP: Linear Explainable Attention in Parallel for causal language modeling with O(1) path length, and O(1) inference
Rushi314/Transformers-for-high-resolution-image-synthesis
Taming Transformers for High-Resolution Image Synthesis