llm-serving
There are 85 repositories under llm-serving topic.
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
ray-project/ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
liguodongiot/llm-action
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
sgl-project/sglang
SGLang is a fast serving framework for large language models and vision language models.
bentoml/OpenLLM
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
skypilot-org/skypilot
SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 15+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
bentoml/BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
superduper-io/superduper
Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.
predibase/lorax
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
microsoft/aici
AICI: Prompts as (Wasm) Programs
MoonshotAI/MoBA
MoBA: Mixture of Block Attention for Long-Context LLMs
ray-project/ray-llm
RayLLM - LLMs on Ray
zhihu/ZhiLight
A highly optimized LLM inference acceleration engine for Llama and its variants.
mosecorg/mosec
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
efeslab/Nanoflow
A throughput-oriented high-performance serving framework for LLMs
alibaba/rtp-llm
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
rohan-paul/LLM-FineTuning-Large-Language-Models
LLM (Large Language Model) FineTuning
hpcaitech/SwiftInfer
Efficient AI Inference & Serving
helixml/helix
🧬 Helix is a private GenAI stack for building AI applications with declarative pipelines, knowledge (RAG), API bindings, and first-class testing.
ray-project/ray-educational-materials
This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.
vllm-project/vllm-ascend
Community maintained hardware plugin for vLLM on Ascend
galeselee/Awesome_LLM_System-PaperList
Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inference acceleration, and related works will be gradually added in the future. Welcome contributions!
substratusai/runbooks
Finetune LLMs on K8s by using Runbooks
torchpipe/torchpipe
Serving Inside Pytorch
HPMLL/BurstGPT
A ChatGPT(GPT-3.5) & GPT-4 Workload Trace to Optimize LLM Serving Systems
chenhunghan/ialacol
🪶 Lightweight OpenAI drop-in replacement for Kubernetes
interestingLSY/swiftLLM
A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).
slai-labs/get-beam
Run GPU inference and training jobs on serverless infrastructure that scales with you.
powerserve-project/PowerServe
High-speed and easy-use LLM serving framework for local deployment
mani-kantap/llm-inference-solutions
A collection of all available inference solutions for the LLMs
asprenger/ray_vllm_inference
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
azminewasi/Awesome-LLMs-ICLR-24
It is a comprehensive resource hub compiling all LLM papers accepted at the International Conference on Learning Representations (ICLR) in 2024.
sugarcane-ai/sugarcane-ai
npm like package ecosystem for Prompts 🤖
bigai-nlco/TokenSwift
From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence Generation
AntonioGr7/pratical-llms
A collection of hand on notebook for LLMs practitioner
friendliai/friendli-client
Friendli: the fastest serving engine for generative AI