linear-attention
There are 12 repositories under linear-attention topic.
BlinkDL/RWKV-LM
RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
lucidrains/taylor-series-linear-attention
Explorations into the recently proposed Taylor Series Linear Attention
lucidrains/agent-attention-pytorch
Implementation of Agent Attention in Pytorch
lironui/Multi-Attention-Network
The semantic segmentation of remote sensing images
lucidrains/autoregressive-linear-attention-cuda
CUDA implementation of autoregressive linear attention, with all the latest research findings
lironui/MAResU-Net
The semantic segmentation of remote sensing images
robflynnyh/hydra-linear-attention
Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)
RWKV-Wiki/rwkv-wiki.github.io
RWKV Wiki website (archived, please visit official wiki)
gmlwns2000/sea-attention
Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)
mtanghu/LEAP
LEAP: Linear Explainable Attention in Parallel for causal language modeling with O(1) path length, and O(1) inference
Rushi314/Transformers-for-high-resolution-image-synthesis
Taming Transformers for High-Resolution Image Synthesis
fattorib/flashy_linear_attention
Flash linear attention kernels in Triton