/SimLLM

Code base for "Sim-LLM: Efficient Inference Optimization Based on Task Similarity for Large Language Models in Edge"

Primary LanguagePython

SimLLM

Codebase for "Sim-LLM: Optimizing LLM Inference at the Edge through Inter-Task KV Reuse"

Evaluation code will be released soon.