Ability to disable or redirect cout/cerr
Opened this issue · 4 comments
Is your feature request related to a problem? Please describe.
The client libraries sometimes use std::cout or std::cerr to send intermediate messages, e.g. in the case of multiple errors in a call, or if a verbose option is enabled. This can lead to garbled output if clients are used in multithreaded contexts.
Describe the solution you'd like
Best solution: some way to swap the default streams for other streams (which may have output disabled, or may be modified to be thread-friendly).
Describe alternatives you've considered
Simpler, but less useful solution: add a preprocessor flag to disable any line with std::cout or std::cerr when building the client libraries. The downside is that some messages would become inaccessible.
Additional context
N/A
I agree that the logging mechanism used by clients is primitive. I would like to instead add an (optional) logger API to the libraries where the user could register a logger object that the library would use for logging. The user could then decide how to handle standard, error and verbose output. That seems like the most general solution. What do you think?
A full-fledged logger API sounds great to me!
The difficulty is that triton uses libraries (e.g. tensorflow and pytorch) that do not provide a logging interface but instead just write directly to stderr and stdout. Triton needs to be able to intercept that output to implement this feature.
My concern here is really just the client libraries, i.e. the uses of cout and cerr in https://github.com/triton-inference-server/client/blob/main/src/c++/library/grpc_client.cc, https://github.com/triton-inference-server/client/blob/main/src/c++/library/http_client.cc.
The backends that write directly to stderr or stdout are running on the server side, so they can't interfere with client-side operations.