Support different network protocols
Closed this issue · 3 comments
Is your feature request related to a problem? Please describe.
In some cases, a server may be deployed and accessed only over an internal, secure network. In this situation, the overhead of TCP could be avoided, reducing latency and improving throughput.
Describe the solution you'd like
Support for other network protocols like UDP or RDMA would be useful. (It's not entirely clear to me how this would interact with gRPC, but ideally it would be nice to select a protocol with the existing gRPC client.)
Describe alternatives you've considered
I'm not sure how else alternative protocols could be used without deploying directly (i.e. without Triton).
We do not have any plans to support any network protocols other than HTTPS and gRPC. However, we do have a C-API. The HTTPS and gRPC protocols are just wrappers around the C-API. You could, in theory, created you own network protocol by wrapping it around C-API.
The C-API does not use any networking, The HTTP/gRPC server implementation use C-API to communicate with the Triton core. See this for more details: https://github.com/triton-inference-server/server/blob/main/docs/images/arch.jpg
We do plan to add the HTTPS support but it may take a while when we get to it.