FMInference

Inference Systems for Foundation Models

Pinned Repositories

DejaVu
Language:Python289 6 3536
FlexiGen
Running large language models on a single GPU for throughput-oriented scenarios.
Language:Python9.2k 112 82548
H2O
[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
Language:Python393 5 3843

FMInference's Repositories

FMInference/FlexiGen
Running large language models on a single GPU for throughput-oriented scenarios.
Language:Python9.2k 112 82548
FMInference/H2O
[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
Language:Python393 5 3843
FMInference/DejaVu
Language:Python289 6 3536