model-serving

There are 147 repositories under model-serving topic.

  • vllm-project/vllm

    A high-throughput and memory-efficient inference and serving engine for LLMs

    Language:Python38.9k3206.7k5.8k
  • BentoML

    bentoml/BentoML

    The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

    Language:Python7.4k751.1k811
  • ahkarami/Deep-Learning-in-Production

    In this repository, I will share some useful notes and references about deploying deep learning-based models in production.

  • kserve/kserve

    Standardized Serverless ML Inference Platform on Kubernetes

    Language:Python3.9k691.9k1.1k
  • FedML-AI/FedML

    FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

    Language:Python3.8k94327744
  • ModelTC/lightllm

    LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

    Language:Python2.9k23193230
  • HuaizhengZhang/AI-System-School

    🚀 Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). 🍻 OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. 🗃️ Llama3, Mistral, etc. 🧑‍💻 Video Tutorials.

  • predibase/lorax

    Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

    Language:Python2.4k31259150
  • tensorchord/envd

    🏕️ Reproducible development environment

    Language:Go2.1k22532159
  • microsoft/aici

    AICI: Prompts as (Wasm) Programs

    Language:Rust2k247682
  • Olares

    beclab/Olares

    Olares: An Open-Source Sovereign Cloud OS for Local AI

    Language:Shell1.8k193762
  • mlrun/mlrun

    MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

    Language:Python1.5k28276257
  • logicalclocks/hopsworks

    Hopsworks - Data-Intensive AI platform with a Feature Store

    Language:Java1.2k3618148
  • truss

    basetenlabs/truss

    The simplest way to serve AI/ML models in production

    Language:Python9511812580
  • zhihu/ZhiLight

    A highly optimized LLM inference acceleration engine for Llama and its variants.

    Language:C++8575112102
  • mosec

    mosecorg/mosec

    A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

    Language:Python8221310161
  • bentoml/Yatai

    Model Deployment at Scale on Kubernetes 🦄️

    Language:TypeScript7921811670
  • efeslab/Nanoflow

    A throughput-oriented high-performance serving framework for LLMs

    Language:Cuda73982829
  • openvinotoolkit/model_server

    A scalable inference server for models optimized with OpenVINO™

    Language:C++70831173216
  • kitops

    jozu-ai/kitops

    An open source DevOps tool for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI artifact.

    Language:Go7071419168
  • alibaba/rtp-llm

    RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

    Language:C++632139654
  • underneathall/pinferencia

    Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

    Language:Python557416787
  • eightBEC/fastapi-ml-skeleton

    FastAPI Skeleton App to serve machine learning models production-ready.

    Language:Python4346384
  • ServerlessLLM

    ServerlessLLM/ServerlessLLM

    Serverless LLM Serving for Everyone.

    Language:Python422128137
  • Lightning-Universe/stable-diffusion-deploy

    Learn to serve Stable Diffusion models on cloud infrastructure at scale. This Lightning App shows load-balancing, orchestrating, pre-provisioning, dynamic batching, GPU-inference, micro-services working together via the Lightning Apps framework.

    Language:Python393195439
  • bentoml/BentoDiffusion

    BentoDiffusion: A collection of diffusion models served with BentoML

    Language:Python35012927
  • AI-Hypercomputer/JetStream

    JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

    Language:Python276202736
  • chitra

    aniketmaurya/chitra

    A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.

    Language:Python22553537
  • lightbend/kafka-with-akka-streams-kafka-streams-tutorial

    Code samples for the Lightbend tutorial on writing microservices with Akka Streams, Kafka Streams, and Kafka

    Language:Scala21131571
  • vllm-project/vllm-ascend

    Community maintained hardware plugin for vLLM on Ascend

    Language:Python16530
  • clearml/clearml-serving

    ClearML - Model-Serving Orchestration and Repository Solution

    Language:Python143115740
  • interestingLSY/swiftLLM

    A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).

    Language:Python1393410
  • spotify/zoltar

    Common library for serving TensorFlow, XGBoost and scikit-learn models in production.

    Language:Java139236733
  • FederatedAI/FATE-Serving

    A scalable, high-performance serving system for federated learning models

    Language:Java138307478
  • bentoml/gallery

    BentoML Example Projects 🎨