Perfomance: circuit-break mechanism
Closed this issue · 1 comments
thegobot commented
The CPU percent(0, 100) ever 1s, as system high water-level, which enable the circuit-break
mechanism, for example, NACK will be disabled if high water-level.
Overwrite by env SRS_CIRCUIT_BREAKER_HIGH_THRESHOLD
for example...what else will be limited?
My server sometimes takes a very long time (6-18 sec.) to process API requests (rtc/v1/whep) and all HTTP requests in general. The total CPU load on the server is no more than 20%
Average SRS process 47% (field cpu_percent from API /v1/summaries, peaks 100% about 6 times a day)
last log for the day where the request time is more than 5 sec.
cat /var/log/nginx/access.log.1 | awk '{print $6 $7 " " $11 " " $13 " " $9 }' | grep whep | awk '{ if($3 > 5) print $0 }'
[01/Sep/2025:08:01:09+0300] 201 8.576 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:09:35:49+0300] 201 6.698 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:12:29:58+0300] 201 7.437 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:13:07:46+0300] 201 8.275 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:13:07:46+0300] 201 7.637 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:13:07:46+0300] 201 7.102 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:13:44:44+0300] 201 8.202 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:14:14:45+0300] 201 6.980 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:14:14:45+0300] 201 6.849 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:14:29:25+0300] 201 7.785 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:14:50:42+0300] 201 5.215 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:15:46:48+0300] 201 9.525 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:15:46:48+0300] 201 9.453 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:15:50:21+0300] 201 6.811 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:15:50:21+0300] 201 6.745 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:15:50:21+0300] 201 5.438 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:15:52:35+0300] 201 8.658 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:17:53:46+0300] 201 7.193 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:17:53:46+0300] 201 7.108 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:17:53:46+0300] 201 7.090 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:17:53:46+0300] 201 6.850 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:17:53:46+0300] 201 6.173 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:18:19:59+0300] 201 7.930 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:18:25:46+0300] 201 6.023 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:19:08:09+0300] 201 7.161 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:19:08:09+0300] 201 5.964 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:19:08:09+0300] 201 5.655 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:19:52:39+0300] 201 7.132 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:19:52:39+0300] 201 6.334 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:19:52:39+0300] 201 6.155 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:20:12:00+0300] 201 7.536 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:20:12:00+0300] 201 7.434 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:20:12:00+0300] 201 5.398 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:20:58:10+0300] 201 8.097 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:20:58:10+0300] 201 6.255 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:21:19:33+0300] 201 9.224 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:21:19:33+0300] 201 8.101 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:21:36:43+0300] 201 6.807 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:21:56:35+0300] 201 5.229 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:22:07:03+0300] 201 6.665 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:22:07:03+0300] 201 6.363 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:22:14:18+0300] 201 5.137 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:22:19:57+0300] 201 11.027 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:22:19:57+0300] 201 8.823 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:22:45:21+0300] 201 5.131 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:22:46:39+0300] 201 8.723 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:06:50+0300] 201 6.566 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:11:53+0300] 201 8.546 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:11:53+0300] 201 7.991 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:11:53+0300] 201 7.938 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:11:53+0300] 201 5.957 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:29:16+0300] 201 5.764 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:29:16+0300] 201 5.445 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:31:40+0300] 201 5.570 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:43:27+0300] 201 5.579 /webrtcs/api/rtc/v1/whep/
[01/Sep/2025:23:57:33+0300] 201 5.092 /webrtcs/api/rtc/v1/whep/
Nginx proxy
location ~ ^/(webrtcs)/api/(.*)$ {
#proxy_read_timeout 3;
proxy_ignore_client_abort on;
proxy_connect_timeout 3s;
proxy_pass http://127.0.0.1:1985/$2$is_args$args;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
}
winlinvip commented
For peak problem, you can increase the pulse to mitigate this.
SRS_CIRCUIT_BREAKER_HIGH_PULSEfrom 2 to 5SRS_CIRCUIT_BREAKER_CRITICAL_PULSEfrom 1 to 10SRS_CIRCUIT_BREAKER_DYING_PULSEto 15