speculative-decoding
There are 33 repositories under speculative-decoding topic.
intel/intel-extension-for-transformers
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
SafeAILab/EAGLE
Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3.
aphrodite-engine/aphrodite-engine
Large-scale LLM inference engine
Infini-AI-Lab/Sequoia
scalable and robust tree-based speculative decoding algorithm
facebookresearch/LayerSkip
Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024
Infini-AI-Lab/TriForce
[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
FasterDecoding/REST
REST: Retrieval-Based Speculative Decoding, NAACL 2024
Infini-AI-Lab/UMbreLLa
LLM Inference on consumer devices
bigai-nlco/TokenSwift
[ICML 2025] |TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation
kssteven418/BigLittleDecoder
[NeurIPS'23] Speculative Decoding with Big Little Decoder
romsto/Speculative-Decoding
Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.
hemingkx/SWIFT
[ICLR 2025] SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
hemingkx/SpecDec
Codes for our paper "Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation" (EMNLP 2023 Findings)
AutonomicPerfectionist/PipeInfer
PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
mscheong01/speculative_decoding.c
minimal C implementation of speculative decoding based on llama2.c
BaohaoLiao/RSD
Reward-guided Speculative Decoding (RSD) for efficiency and effectiveness.
jadohu/LANTERN
Official Implementation of LANTERN (ICLR'25) and LANTERN++(ICLRW-SCOPE'25)
Geralt-Targaryen/Awesome-Speculative-Decoding
Reading notes on Speculative Decoding papers
ccs96307/fast-llm-inference
Accelerating LLM inference with techniques like speculative decoding, quantization, and kernel fusion, focusing on implementing state-of-the-art research papers.
hsj576/GRIFFIN
Official Implementation of "GRIFFIN: Effective Token Alignment for Faster Speculative Decoding"
smpanaro/token-recycling
Unofficial implementation of Token Recycling self-speculative decoding method.
wtlow003/speculative-sampling
Implementation of Speculative Sampling in "Accelerating Large Language Model Decoding with Speculative Sampling"
jayeshthk/SpS-SpecDec
SpS-SpecDec: a fast Python lib that boosts autoregressive LM inference with speculative decoding. Inspired by DeepMind, it guesses multiple tokens using a small draft model, verifies with a big one. Get 2-2.5x speedups, no quality drop!
pinqian77/Dynasurge
Dynasurge: Dynamic Tree Speculation for Prompt-Specific Decoding
PopoDev/BiLD
Reproducibility Project for [NeurIPS'23] Speculative Decoding with Big Little Decoder
u-hyszk/japanese-speculative-decoding
Verification of the effect of speculative decoding in Japanese.
haukzero/Speculative-Demo
一个简单的投机推理实现
jayeshthk/speculative-decoding-inference
Speculative decoding challenge by anysphere(cursor AI).
natask/infra_gpu_hack
A novel algorithm that integrates a text, diffusion LLM as a draft model to boost the performance of traditional auto-regressive LLMs.
realjules/align_llm
The LLM Defense Framework enhances large language model security through post-processing defenses and statistical guarantees based on one-class SVM. It combines advanced sampling methods with adaptive policy updates and comprehensive evaluation metrics, providing researchers and practitioners with tools to build more secure AI systems.
kinshukdua/SpecDec
Some experiments aimed at increasing LLM throughput and efficiency via Speculative Decoding.
majid-daliri/DISD
Coupling without Communication and Drafter-Invariant Speculative Decoding
wtlow003/ngram-decoding
(Re)-implementation of "Prompt Lookup Decoding" by Apoorv Saxena, with extended ideas from LLMA Decoding.