inference
There are 1186 repositories under inference topic.
hpcaitech/ColossalAI
Making large AI models cheaper, faster and more accessible
microsoft/DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
ggerganov/whisper.cpp
Port of OpenAI's Whisper model in C/C++
google-ai-edge/mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
Tencent/ncnn
ncnn is a high-performance neural network inference framework optimized for the mobile platform
gvergnaud/ts-pattern
🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.
aws/amazon-sagemaker-examples
Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
SYSTRAN/faster-whisper
Faster Whisper transcription with CTranslate2
NVIDIA/TensorRT
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
huggingface/text-generation-inference
Large Language Model Text Generation Inference
triton-inference-server/server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
dusty-nv/jetson-inference
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB
💎1MB lightweight face detection model (1MB轻量级人脸检测模型)
gcanti/io-ts
Runtime type system for IO decoding/encoding
openvinotoolkit/openvino
OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
Trusted-AI/adversarial-robustness-toolbox
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
SuperDuperDB/superduperdb
🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.
NVIDIA-AI-IOT/torch2trt
An easy to use PyTorch to TensorRT converter
Tencent/TNN
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.
openvinotoolkit/open_model_zoo
Pre-trained Deep Learning models and demos (high quality and extremely fast)
AutoGPTQ/AutoGPTQ
An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
vaticle/typedb
TypeDB: the polymorphic database powered by types
bytedance/lightseq
LightSeq: A High Performance Library for Sequence Processing and Generation
xorbitsai/inference
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
neuralmagic/deepsparse
Sparsity-aware deep learning inference runtime for CPUs
OpenNMT/CTranslate2
Fast inference engine for Transformer models
tencentmusic/cube-studio
cube studio开源云原生一站式机器学习/深度学习AI平台，支持sso登录，多租户/多项目组，大数据平台对接，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU，边缘计算，serverless，标注平台，自动化标注，数据集管理，大模型微调，vllm大模型推理，llmops，私有知识库，AI模型应用商店，支持模型一键开发/推理/微调，支持国产cpu/gpu/npu芯片，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/spark/ray/volcano分布式
pgmpy/pgmpy
Python Library for learning (Structure and Parameter), inference (Probabilistic and Causal), and simulations in Bayesian Networks.
argmaxinc/WhisperKit
Swift native on-device speech recognition with Whisper for Apple Silicon
HuaizhengZhang/Awesome-System-for-Machine-Learning
A curated list of research in machine learning systems (MLSys). Paper notes are also provided.
huggingface/optimum
🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools
zjhellofss/KuiperInfer
带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step
openvinotoolkit/openvino_notebooks
📚 Jupyter notebook tutorials for OpenVINO™
tairov/llama2.mojo
Inference Llama 2 in one file of pure 🔥
tobegit3hub/tensorflow_template_application
TensorFlow template application for deep learning