awesome-long-context Efficient Inference, Sparse Attention, Efficient KV Cache [2020/01] Reformer: The Efficient Transformer [2020/06] Linformer: Self-Attention with Linear Complexity [2022/12] Parallel Context Windows for Large Language Models [2023/04] Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering [2023/05] Landmark Attention: Random-Access Infinite Context Length for Transformers [2023/05] Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time [2023/06] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time [2023/06] H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models [2023/07] Scaling In-Context Demonstrations with Structured Attention [2023/08] LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models [2023/09] EFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKS [2023/10] HyperAttention: Long-context Attention in Near-Linear Time [2023/10] TRAMS: Training-free Memory Selection for Long-range Language Modeling External Memory & Information Retrieval [2023/06] Augmenting Language Models with Long-Term Memory [2023/06] Long-range Language Modeling with Self-retrieval [2023/07] Focused Transformer: Contrastive Training for Context Scaling Positional Encoding [2021/04] RoFormer: Enhanced Transformer with Rotary Position Embedding [2022/03] Transformer Language Models without Positional Encodings Still Learn Positional Information [2022/04] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation [2022/05] KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation [2022/12] Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis [2022/12] The Impact of Positional Encoding on Length Generalization in Transformers [2023/05] Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings [2023/06] Extending Context Window of Large Language Models via Positional Interpolation [2023/07] Exploring Transformer Extrapolation [2023/09] YaRN: Efficient Context Window Extension of Large Language Models [2023/09] Effective Long-Context Scaling of Foundation Models [2023/10] CLEX: Continuous Length Extrapolation for Large Language Models Context Compression [2022/12] Structured Prompting: Scaling In-Context Learning to 1,000 Examples [2023/05] Efficient Prompting via Dynamic In-Context Learning [2023/05] Adapting Language Models to Compress Contexts [2023/05] Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers [2023/07] In-context Autoencoder for Context Compression in a Large Language Model [2023/10] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs [2023/10] RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation [2023/10] Compressing Context to Enhance Inference Efficiency of Large Language Models [2023/10] LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models [2023/10] LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression [2023/10] TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction Architecture Variances [2021/11] Efficiently Modeling Long Sequences with Structured State Spaces [2022/12] Hungry Hungry Hippos: Towards Language Modeling with State Space Models [2023/02] Hyena Hierarchy: Towards Larger Convolutional Language Models [2023/04] Scaling Transformer to 1M tokens and beyond with RMT [2023/06] Block-State Transformer [2023/07] Retentive Network: A Successor to Transformer for Large Language Models [2023/10] Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors [2023/10] Mamba: Linear-Time Sequence Modeling with Selective State Spaces White-Box [2019/06] Theoretical Limitations of Self-Attention in Neural Sequence Models [2020/06] $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers [2022/02] Overcoming a Theoretical Limitation of Self-Attention [2023/05] Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer [2023/10] JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention Long Context Modeling [2023/07] LongNet: Scaling Transformers to 1,000,000,000 Tokens [2023/08] Giraffe: Adventures in Expanding Context Lengths in LLMs [2023/09] LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models [2023/10] Mistral 7B Benchmarks [2020/11] Long Range Arena: A Benchmark for Efficient Transformers [2022/01] SCROLLS: Standardized CompaRison Over Long Language Sequences [2023/01] LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization [2023/05] ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding [2023/08] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding [2023/10] M4LE: A MULTI-ABILITY MULTI-RANGE MULTITASK MULTI-DOMAIN LONG-CONTEXT EVALUATION BENCHMARK FOR LARGE LANGUAGE MODELS Data [2023/12] Structured Packing in LLM Training Improves Long Context Utilization [2024/01] LongAlign: A Recipe for Long Context Alignment of Large Language Models [2024/02] Data Engineering for Scaling Language Models to 128K Context Others [2023/07] Zero-th Order Algorithm for Softmax Attention Optimization [2023/10] (Dynamic) Prompting might be all you need to repair Compressed LLMs [2023/10] Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
