triton-inference-server/hugectr_backend

How to turn off only the request log? (The request log is too verbose, resulting in poor throughput)

kisow opened this issue · 3 comments

kisow commented

The more detailed the server startup log, the better it will prevent mistakes or help debugging.
However, if the request log becomes too verbose, it leads to poor server throughput performance.
In fact, turning on the info log alone cuts the server throughput in half for the same 40 thread request.

How can I make it quiet by turning off only the request log while leaving the info log on when the server starts?

Below is the client-side benchmark result for the same request.

--log-info=false

This leaves no lines in the server log.

[Q0000041000][T12] 2282.60QPS avg.etl/infer/total:  3.87ms  10.00ms  15.20ms top.etl/infer/total:  9.00ms  15.00ms  22.00ms
[Q0000042000][T10] 2281.58QPS avg.etl/infer/total:  3.69ms  11.43ms  16.29ms top.etl/infer/total:  8.00ms  17.00ms  25.00ms
[Q0000043000][T11] 2282.70QPS avg.etl/infer/total:  4.13ms  11.52ms  16.87ms top.etl/infer/total: 10.00ms  21.00ms  28.00ms
[Q0000044000][T14] 2281.04QPS avg.etl/infer/total:  4.10ms  10.98ms  16.33ms top.etl/infer/total:  9.00ms  18.00ms  26.00ms
[Q0000045000][T33] 2280.70QPS avg.etl/infer/total:  3.43ms  11.81ms  16.51ms top.etl/infer/total:  8.00ms  19.00ms  24.00ms
[Q0000046000][T03] 2279.96QPS avg.etl/infer/total:  4.07ms   9.99ms  15.36ms top.etl/infer/total: 10.00ms  16.00ms  23.00ms

--log-info=true

This option makes the server request log too verbose, cutting throughput in half.

[Q0000041000][T11] 1108.63QPS avg.etl/infer/total:  3.89ms  37.53ms  42.71ms top.etl/infer/total:  9.00ms  69.00ms  77.00ms
[Q0000042000][T26] 1108.04QPS avg.etl/infer/total:  4.26ms  22.60ms  28.14ms top.etl/infer/total: 10.00ms  33.00ms  40.00ms
[Q0000043000][T37] 1107.30QPS avg.etl/infer/total:  3.88ms  23.35ms  28.63ms top.etl/infer/total: 10.00ms  39.00ms  45.00ms
[Q0000044000][T19] 1109.48QPS avg.etl/infer/total:  4.11ms  23.69ms  29.10ms top.etl/infer/total: 10.00ms  35.00ms  42.00ms
[Q0000045000][T17] 1107.12QPS avg.etl/infer/total:  3.81ms  41.57ms  46.75ms top.etl/infer/total:  9.00ms  75.00ms  81.00ms
[Q0000046000][T27] 1108.08QPS avg.etl/infer/total:  4.35ms  24.15ms  29.93ms top.etl/infer/total: 14.00ms  32.00ms  43.00ms
yingcanw commented

This issue is related to the log control logic of triton server, HugeCTR Backend cannot change the log level at runtime.

yingcanw commented

@kisow We have fixed this issue in the next release 22.06.
The Triton request log is off by default(Hugectr backend and Triton initialization log as info level will still be output ).
If you want to turn on the request log, just add "--log-verbose=true" when launching Triton Server.
If you want to turn on the ps request log, just add env variable HUGECTR_LOG_LEVEL=9 before launching Triton Server(trace level log output)

  • 👍1
kisow commented

I have tested the released 22.06 and checked out that only the request log can be turned off while leaving the server boot up log in detail. thank you :)