inference
There are 1648 repositories under inference topic.
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
ggml-org/whisper.cpp
Port of OpenAI's Whisper model in C/C++
hpcaitech/ColossalAI
Making large AI models cheaper, faster and more accessible
deepspeedai/DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
google-ai-edge/mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
Tencent/ncnn
ncnn is a high-performance neural network inference framework optimized for the mobile platform
SYSTRAN/faster-whisper
Faster Whisper transcription with CTranslate2
sgl-project/sglang
SGLang is a fast serving framework for large language models and vision language models.
stas00/ml-engineering
Machine Learning Engineering Open Book
gvergnaud/ts-pattern
🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.
NVIDIA/TensorRT
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
aws/amazon-sagemaker-examples
Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
huggingface/text-generation-inference
Large Language Model Text Generation Inference
triton-inference-server/server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
openvinotoolkit/openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
xorbitsai/inference
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
dusty-nv/jetson-inference
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
oumi-ai/oumi
Easily fine-tune, evaluate and deploy gpt-oss, Qwen3, DeepSeek-R1, or any open source LLM / VLM!
Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB
💎1MB lightweight face detection model (1MB轻量级人脸检测模型)
gcanti/io-ts
Runtime type system for IO decoding/encoding
GeeeekExplorer/nano-vllm
Nano vLLM
Trusted-AI/adversarial-robustness-toolbox
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
LMCache/LMCache
Supercharge Your LLM with the Fastest KV Cache Layer
superduper-io/superduper
Superduper: End-to-end framework for building custom AI applications and agents.
argmaxinc/WhisperKit
On-device Speech Recognition for Apple Silicon
AutoGPTQ/AutoGPTQ
An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
NVIDIA-AI-IOT/torch2trt
An easy to use PyTorch to TensorRT converter
Tencent/TNN
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.
tencentmusic/cube-studio
cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式
openvinotoolkit/open_model_zoo
Pre-trained Deep Learning models and demos (high quality and extremely fast)
typedb/typedb
TypeDB: the power of programming, in your database
OpenNMT/CTranslate2
Fast inference engine for Transformer models
OpenCSGs/csghub
CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! ⭐️
kvcache-ai/Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
gpustack/gpustack
Simple, scalable AI model deployment on GPU clusters
PaddlePaddle/FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle