serving
There are 122 repositories under serving topic.
ray-project/ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
tensorflow/serving
A flexible, high-performance serving system for machine learning models
volcano-sh/volcano
A Cloud Native Batch System (Project under CNCF)
SeldonIO/seldon-core
An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
ahkarami/Deep-Learning-in-Production
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
pytorch/serve
Serve, optimize and scale PyTorch models in production
Lightning-AI/LitServe
The easiest way to deploy agents, MCP servers, models, RAG, pipelines and more. No MLOps. No YAML.
PaddlePaddle/FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
skyzh/tiny-llm
A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny vLLM + Qwen.
georgia-tech-db/evadb
Database system for AI-powered apps
tobegit3hub/tensorflow_template_application
TensorFlow template application for deep learning
ray-project/llm-applications
A comprehensive guide to building RAG-based LLM applications for production.
dingodb/dingo
A multi-modal vector database that supports upserts and vector queries using unified SQL (MySQL-Compatible) on structured and unstructured data, while meeting the requirements of high concurrency and ultra-low latency.
Delta-ML/delta
DELTA is a deep learning based natural language and speech processing platform. LF AI & DATA Projects: https://lfaidata.foundation/projects/delta/
PaddlePaddle/Serving
A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）
openvinotoolkit/model_server
A scalable inference server for models optimized with OpenVINO™
tobegit3hub/simple_tensorflow_serving
Generic and easy-to-use serving service for machine learning models
underneathall/pinferencia
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
meta-soul/MetaSpore
A unified end-to-end machine intelligence platform
vectorch-ai/ScaleLLM
A high-performance inference system for large language models, designed for production environments.
polyaxon/haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
zzsza/Boostcamp-AI-Tech-Product-Serving
부스트캠프 AI Tech - Product Serving 자료
bodywork-ml/bodywork-core
ML pipeline orchestration and model deployments on Kubernetes.
Hydrospheredata/hydro-serving
MLOps Platform
deepjavalibrary/djl-serving
A universal scalable machine learning model deployment solution
outcaste-io/outserv
Blockchain Search with GraphQL APIs
cap-ntu/ML-Model-CI
MLModelCI is a complete MLOps platform for managing, converting, profiling, and deploying MLaaS (Machine Learning-as-a-Service), bridging the gap between current ML training and serving systems.
NetEase-Media/grps
Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.
torchpipe/torchpipe
Serving Inside Pytorch
clearml/clearml-serving
ClearML - Model-Serving Orchestration and Repository Solution
krystianity/keras-serving
bring keras-models to production with tensorflow-serving and nodejs + docker :pizza:
emacski/tensorflow-serving-arm
TensorFlow Serving ARM - A project for cross-compiling TensorFlow Serving targeting popular ARM cores
notAI-tech/fastDeploy
Deploy DL/ ML inference pipelines with minimal extra code.
AI-Hypercomputer/gpu-recipes
Recipes for reproducing training and serving benchmarks for large machine learning models using GPUs on Google Cloud.
balavenkatesh3322/model_deployment
A collection of model deployment library and technique.