triton-inference-server/server

Support different network protocols

Closed this issue · 3 comments

kpedro88 commented

Is your feature request related to a problem? Please describe.
In some cases, a server may be deployed and accessed only over an internal, secure network. In this situation, the overhead of TCP could be avoided, reducing latency and improving throughput.

Describe the solution you'd like
Support for other network protocols like UDP or RDMA would be useful. (It's not entirely clear to me how this would interact with gRPC, but ideally it would be nice to select a protocol with the existing gRPC client.)

Describe alternatives you've considered
I'm not sure how else alternative protocols could be used without deploying directly (i.e. without Triton).

dzier commented

We do not have any plans to support any network protocols other than HTTPS and gRPC. However, we do have a C-API. The HTTPS and gRPC protocols are just wrappers around the C-API. You could, in theory, created you own network protocol by wrapping it around C-API.

okyspace commented

@dzier does triton support https as of now? I do not find the codes in C-API.

tanmayv25 commented

The C-API does not use any networking, The HTTP/gRPC server implementation use C-API to communicate with the Triton core. See this for more details: https://github.com/triton-inference-server/server/blob/main/docs/images/arch.jpg

We do plan to add the HTTPS support but it may take a while when we get to it.