windyuan/triton-inference-server

The Triton Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs.

NVIDIA Triton Inference Server

NEW NAME: We have a new name: Triton Inference Server. Read about why we are making this change and our plans for version 2 of the inference server in Roadmap.

LATEST RELEASE: You are currently on the master branch which tracks under-development progress towards the next release. The latest release of the Triton Inference Server is 1.12.0 and is available on branch r20.03.

NVIDIA Triton Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton Server is also available as a shared library with an API that allows the full functionality of the server to be included directly in an application. Triton Server provides the following features:

• Multiple framework support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel, ONNX, PyTorch, and Caffe2 NetDef model formats. Also supports TensorFlow-TensorRT and ONNX-TensorRT integrated models. Variable-size input and output tensors are allowed if supported by the framework. See Capabilities for detailed support information for each framework.
• Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.
• Batching support. For models that support batching, Triton Server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. Triton Server also supports multiple scheduling and batching algorithms that combine individual inference requests together to improve inference throughput. These scheduling and batching decisions are transparent to the client requesting inference.
• Custom backend support. Triton Server allows individual models to be implemented with custom backends instead of by a deep-learning framework. With a custom backend a model can implement any logic desired, while still benefiting from the GPU support, concurrent execution, dynamic batching and other features provided by the server.
• Ensemble support. An ensemble represents a pipeline of one or more models and the connection of input and output tensors between those models. A single inference request to an ensemble will trigger the execution of the entire pipeline.
• Multi-GPU support. Triton Server can distribute inferencing across all system GPUs.
• Triton Server provides multiple modes for model management. These model management modes allow for both implicit and explicit loading and unloading of models without requiring a server restart.
• Model repositories may reside on a locally accessible file system (e.g. NFS), in Google Cloud Storage or in Amazon S3.
• Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.
• Metrics indicating GPU utilization, server throughput, and server latency.
• C library inferface allows the full functionality of Triton Server to be included directly in an application.

The current release of the Triton Inference Server is 1.12.0 and corresponds to the 20.02 release of the tensorrtserver container on NVIDIA GPU Cloud (NGC). The branch for this release is r20.03.

Backwards Compatibility

Continuing in the latest version the following interfaces maintain backwards compatibilty with the 1.0.0 release. If you have model configuration files, custom backends, or clients that use the inference server HTTP or GRPC APIs (either directly or through the client libraries) from releases prior to 1.0.0 you should edit and rebuild those as necessary to match the version 1.0.0 APIs.

The following inferfaces will maintain backwards compatibility for all future 1.x.y releases (see below for exceptions):

• Model configuration as defined in model_config.proto.
• The inference server HTTP and GRPC APIs as defined in api.proto and grpc_service.proto, except as noted below.
• The V1 and V2 custom backend interfaces as defined in custom.h.

As new features are introduced they may temporarily have beta status where they are subject to change in non-backwards-compatible ways. When they exit beta they will conform to the backwards-compatibility guarantees described above. Currently the following features are in beta:

• The inference server library API as defined in trtserver.h is currently in beta and may undergo non-backwards-compatible changes.
• The C++ and Python client libraries are not stictly included in the inference server compatibility guarantees and so should be considered as beta status.

Roadmap

The inference server's new name is Triton Inference Server, which can be shortened to just Triton Server in contexts where inferencing is already understood. The primary reasons for the name change are to :

• Avoid confusion with the NVIDIA TensorRT Programmable Inference Accelerator.
• Avoid the perception that Triton Server only supports TensorRT models when in fact the server supports a wide range of model frameworks and formats.
• Highlight that the server is aligning HTTP/REST and GRPC protocols with a set of KFServing community standard inference protocols that have been proposed by the KFServing project.

Transitioning from the current protocols (version 1) to the new protocols (version 2) will take place over several releases.

• Current master
  • Alpha release of server support for KFServing community standard GRPC and HTTP/REST inference protocol.
  • Alpha release of Python client library that uses KFServing community standard GRPC and HTTP/REST inference protocol.
  • See client documentation for description and examples showing how to enable and use the new GRPC and HTTP/REST inference protocol and Python client library.
  • Existing HTTP/REST and GRPC protocols, and existing client APIs continue to be supported and remain the default protocols.
• 20.05
  • Beta release of KFServing community standard HTTP/REST and GRPC inference protocol support in server, Python client, and C++ client.
  • Beta release of the HTTP/REST and GRPC extensions to the KFServing inference protocol.
  • Existing HTTP/REST and GRPC protocols are deprecated but remain the default.
  • Existing shared library inferface defined in trtserver.h continues to be supported but is deprecated.
  • Beta release of new shared library interface is defined in tritonserver.h.
• 20.06
  • Triton Server version 2.0.0.
  • KFserving community standard HTTP/REST and GRPC inference protocols plus all Triton extensions become the default and only supported protocols for the server.
  • C++ and Python client libraries based on the KFServing standard inference protocols become the default and only supported client libraries.
  • The new shared library interface defined in tritonserver.h becomes the default and only supported shared library interface.
  • Original C++ and Python client libraries are removed. Release 20.05 is the last release to support these libraries.
  • Original shared library interface defined in trtserver.h is removed. Release 20.05 is the last release to support the trtserver.h shared library interface.

Throughout the transition the model repository struture and custom backend APIs will remain unchanged so that any existing model repository and custom backends will continue to work with Triton Server.

In the 20.06 release there will be some minor changes to the tritonserver command-line executable arguments. It will be necessary to revisit and possible adjust invocations of tritonserver executable.

In the 20.06 release there will be some minor changes to the model configuration schema. It is expected that these changes will not impact the vast majority of model configurations. For impacted models the model configuration will need minor edits to become compatible with Triton Server version 2.0.0.

Documentation

The User Guide, Developer Guide, and API Reference documentation for the current release provide guidance on installing, building, and running Triton Inference Server.

You can also view the documentation for the master branch and for earlier releases.

An FAQ provides answers for frequently asked questions.

READMEs for deployment examples can be found in subdirectories of deploy/, for example, deploy/single_server/README.rst.

The Release Notes and Support Matrix indicate the required versions of the NVIDIA Driver and CUDA, and also describe which GPUs are supported by Triton Server.

Presentations and Papers

• High-Performance Inferencing at Scale Using the TensorRT Inference Server.
• Accelerate and Autoscale Deep Learning Inference on GPUs with KFServing.
• Deep into Triton Inference Server: BERT Practical Deployment on NVIDIA GPU.
• Maximizing Utilization for Data Center Inference with TensorRT Inference Server.
• NVIDIA TensorRT Inference Server Boosts Deep Learning Inference.
• GPU-Accelerated Inference for Kubernetes with the NVIDIA TensorRT Inference Server and Kubeflow.

Contributing

Contributions to Triton Inference Server are more than welcome. To contribute make a pull request and follow the guidelines outlined in the Contributing document.

Reporting problems, asking questions

We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are:

• minimal – use as little code as possible that still produces the same problem
• complete – provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing problems the more time we have to fix it
• verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question.