How to modify the value of nginx-unit backlog parameter? I want to increase it to meet my performance needs. It's time to change the default value, 511 is no longer applicable
Closed this issue · 26 comments
Reasons for asking this question: I designed a web server test that server answers as fast as possible But Windows was dropping a few requests even if server answers as fast as possible. Because Clients disconnected when backlog was full , This is because modern servers fetch more than 511 resources at a time and nginx/php-fpm backlog ( #define NGX_LISTEN_BACKLOG 511) constant was hard coded to 511. Then I Calling listen(2)'s backlog parameter in server with a queue size larger than 511 solved the issue
And The rate at which new connections can be accepted is equal to the number of entries which can fit on the listen queue divided by the average length of time each entry spends on the queue. Therefore, the larger the queue, the greater the rate at which new connection requests can be accepted. so how to Adjusting the connection backlog size? like php-fpm (listen.backlog or nginx ip.80 backlog=xxx) can be setting, i canot find this setting in nginx united configure
Before adjustment：
wrk -t1 -c2500 -d10s --latency http://127.0.0.1:8001/
Running 10s test @ http://127.0.0.1:8001/
1 threads and 2500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 181.19ms 23.27ms 1.63s 94.81%
Req/Sec 4.82k 537.57 7.80k 92.86%
Latency Distribution
50% 181.75ms
75% 185.59ms
90% 190.95ms
99% 229.68ms
47011 requests in 10.05s, 161.98MB read
Socket errors: connect 0, read 2038, write 598, timeout 28
Requests/sec: 4676.27
Transfer/sec: 16.11MB
After adjustment：
wrk -t1 -c2500 -d10s --latency http://127.0.0.1:8787/
Running 10s test @ http://127.0.0.1:8787/
1 threads and 2500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 17.44ms 7.58ms 48.34ms 59.73%
Req/Sec 73.81k 14.25k 99.75k 69.41%
Latency Distribution
50% 17.43ms
75% 23.72ms
90% 27.35ms
99% 33.11ms
667467 requests in 10.07s, 1.18GB read
Requests/sec: 66292.37
Transfer/sec: 119.68MB
The above is my test. Backlog is very important, but I couldn't find where to configure in NGINX Unit ?
Just to check, are you asking about NGINX Unit or nginx the web server?
Just to check, are you asking about NGINX Unit or nginx the web server?
NGINX Unit
I take it you're using Unit on Linux?. As you see we hardcode the listen(2) backlog to 511, also on Linux the maximum you can set this to is controlled via /proc/sys/net/core/somaxconn which on my system is 4096, so quite a bit room to spare...
Currently there is no way to adjust this other than changing
#define NXT_LISTEN_BACKLOG 511 in src/nxt_listen_socket.h.
However I see no real reason why we couldn't make it configurable...
Are you referring to the inability to configure through listeners, routes, and applications?Can we only manually recompile src/nxt_listen_stocket. h, right
Can we add such a configuration?
Can we add such a configuration?
We'll discuss it with the team. "backlog" looks like more concise.
{
"*:8001": {
"pass": "routes",
"backlog": 4096
}
}
Btw, how does "backlog" apply to "processes"?
Thank you for the team's efforts. I am a developer from China. There are 700M people using nginx products here. if ok I will promote it.
processes
The above is a conjecture，I think this should be bound to listener
Can we add such a configuration?
I don't think the listen backlog has any relevance to "applications". The router process does the listen(2) not the application processes...
Btw, how does "backlog" apply to "processes"?
Not sure I understand, the listen backlog is the second parameter to the listen(2) system-call.
SYNOPSIS
#include <sys/socket.h>
int listen(int sockfd, int backlog);
It's a per-socket thing.
Feel free to give the following patch a spin... (note: It doesn't effect the control socket)
diff --git ./src/nxt_conf_validation.c ./src/nxt_conf_validation.c
index 04091745..267a897d 100644
--- ./src/nxt_conf_validation.c
+++ ./src/nxt_conf_validation.c
@@ -176,6 +176,8 @@ static nxt_int_t nxt_conf_vldt_app_name(nxt_conf_validation_t *vldt,
nxt_conf_value_t *value, void *data);
static nxt_int_t nxt_conf_vldt_forwarded(nxt_conf_validation_t *vldt,
nxt_conf_value_t *value, void *data);
+static nxt_int_t nxt_conf_vldt_listen_backlog(nxt_conf_validation_t *vldt,
+ nxt_conf_value_t *value, void *data);
static nxt_int_t nxt_conf_vldt_app(nxt_conf_validation_t *vldt,
nxt_str_t *name, nxt_conf_value_t *value);
static nxt_int_t nxt_conf_vldt_object(nxt_conf_validation_t *vldt,
@@ -424,6 +426,10 @@ static nxt_conf_vldt_object_t nxt_conf_vldt_listener_members[] = {
.type = NXT_CONF_VLDT_OBJECT,
.validator = nxt_conf_vldt_object,
.u.members = nxt_conf_vldt_client_ip_members
+ }, {
+ .name = nxt_string("backlog"),
+ .type = NXT_CONF_VLDT_NUMBER,
+ .validator = nxt_conf_vldt_listen_backlog,
},
#if (NXT_TLS)
@@ -2677,6 +2683,28 @@ nxt_conf_vldt_forwarded(nxt_conf_validation_t *vldt, nxt_conf_value_t *value,
}
+static nxt_int_t
+nxt_conf_vldt_listen_backlog(nxt_conf_validation_t *vldt,
+ nxt_conf_value_t *value, void *data)
+{
+ int64_t backlog;
+
+ backlog = nxt_conf_get_number(value);
+
+ if (backlog < 1) {
+ return nxt_conf_vldt_error(vldt, "The \"backlog\" number must be "
+ "equal to or greater than 1.");
+ }
+
+ if (backlog > NXT_INT32_T_MAX) {
+ return nxt_conf_vldt_error(vldt, "The \"backlog\" number must "
+ "not exceed %d.", NXT_INT32_T_MAX);
+ }
+
+ return NXT_OK;
+}
+
+
static nxt_int_t
nxt_conf_vldt_app(nxt_conf_validation_t *vldt, nxt_str_t *name,
nxt_conf_value_t *value)
diff --git ./src/nxt_router.c ./src/nxt_router.c
index 43209451..5842bcff 100644
--- ./src/nxt_router.c
+++ ./src/nxt_router.c
@@ -166,7 +166,7 @@ static void nxt_router_app_prefork_ready(nxt_task_t *task,
static void nxt_router_app_prefork_error(nxt_task_t *task,
nxt_port_recv_msg_t *msg, void *data);
static nxt_socket_conf_t *nxt_router_socket_conf(nxt_task_t *task,
- nxt_router_temp_conf_t *tmcf, nxt_str_t *name);
+ nxt_router_temp_conf_t *tmcf, nxt_str_t *name, int backlog);
static nxt_int_t nxt_router_listen_socket_find(nxt_router_temp_conf_t *tmcf,
nxt_socket_conf_t *nskcf, nxt_sockaddr_t *sa);
@@ -1959,12 +1959,22 @@ nxt_router_conf_create(nxt_task_t *task, nxt_router_temp_conf_t *tmcf,
next = 0;
for ( ;; ) {
+ int listen_backlog = 0;
+ nxt_conf_value_t *backlog;
+
+ static const nxt_str_t backlog_path = nxt_string("backlog");
+
listener = nxt_conf_next_object_member(listeners, &name, &next);
if (listener == NULL) {
break;
}
- skcf = nxt_router_socket_conf(task, tmcf, &name);
+ backlog = nxt_conf_get_object_member(listener, &backlog_path, NULL);
+ if (backlog != NULL) {
+ listen_backlog = nxt_conf_get_number(backlog);
+ }
+
+ skcf = nxt_router_socket_conf(task, tmcf, &name, listen_backlog);
if (skcf == NULL) {
goto fail;
}
@@ -2684,7 +2694,7 @@ nxt_router_application_init(nxt_router_conf_t *rtcf, nxt_str_t *name,
static nxt_socket_conf_t *
nxt_router_socket_conf(nxt_task_t *task, nxt_router_temp_conf_t *tmcf,
- nxt_str_t *name)
+ nxt_str_t *name, int backlog)
{
size_t size;
nxt_int_t ret;
@@ -2728,7 +2738,7 @@ nxt_router_socket_conf(nxt_task_t *task, nxt_router_temp_conf_t *tmcf,
nxt_listen_socket_remote_size(ls);
ls->socket = -1;
- ls->backlog = NXT_LISTEN_BACKLOG;
+ ls->backlog = backlog > 0 ? backlog : NXT_LISTEN_BACKLOG;
ls->flags = NXT_NONBLOCK;
ls->read_after_accept = 1;
}
@@ -2875,7 +2885,7 @@ nxt_router_listen_socket_ready(nxt_task_t *task, nxt_port_recv_msg_t *msg,
nxt_socket_defer_accept(task, s, rpc->socket_conf->listen->sockaddr);
- ret = nxt_listen_socket(task, s, NXT_LISTEN_BACKLOG);
+ ret = nxt_listen_socket(task, s, rpc->socket_conf->listen->backlog);
if (nxt_slow_path(ret != NXT_OK)) {
goto fail;
}E.g
"listeners": {
"[::1]:8080": {
"pass": "routes",
"backlog": 1024
},
"127.0.0.1:8080": {
"pass": "routes"
}
},$ ss -tunxlp | grep unit
u_str LISTEN 0 511 /opt/unit/control.unit.sock.tmp 4302333 * 0 users:(("unitd",pid=18290,fd=6),("unitd",pid=18289,fd=6),("unitd",pid=18287,fd=6))
tcp LISTEN 0 511 127.0.0.1:8080 0.0.0.0:* users:(("unitd",pid=18290,fd=12))
tcp LISTEN 0 1024 [::1]:8080 [::]:* users:(("unitd",pid=18290,fd=11))
One socket "::1" set to a backlog of 1024 the other "127.0.0.1" not set, using the default and the control socket just using the default.
Great, really amazing brother, but I don't know C syntax and how to recompile and build nginx unit. Can you release the entire new and complete version? I will continue to test it
How to provide a video tutorial and guidance so that people who don't know C language can build and compile their own nginx unit projects? It's definitely the most perfect. By the way, I am a PHP developer, and it's very practical but the threshold is too low, so many people don't know how to use C at a lower level (including me). However, I believe PHP is the best web language, and 80% of Chinese developers also think so. There are about 2 million PHP developers in China
OK, no problem. Hopefully we can get this into the next release which is happening Real Soon Now(tm).
Hmm, just something interesting...
In src/nxt_listen_socket.h we have
#if (NXT_FREEBSD || NXT_MACOSX || NXT_OPENBSD)
/*
* A backlog is limited by system-wide sysctl kern.ipc.somaxconn.
* This is supported by FreeBSD 2.2, OpenBSD 2.0, and MacOSX.
*/
#define NXT_LISTEN_BACKLOG -1
#else
/*
* Linux, Solaris, and NetBSD treat negative value as 0.
* 511 is a safe default.
*/
#define NXT_LISTEN_BACKLOG 511
#endif However on at least Linux 2.6.12-rc5 (as far back the main repo goes and I very much doubt we care about anything even remotely that old), Linux does not treat a negative value as 0, it does the same as the BSDs and uses the setting of /proc/sys/net/core/somaxconn with this Kernel code in net/socket.c
int __sys_listen(int fd, int backlog)
{
...
if ((unsigned int)backlog > somaxconn)
backlog = somaxconn;
...
}We pass in an int (which is signed). We cast it to an unsigned int, unsigned integer over/underflow is defined behaviour and so a value of -1 for example would wrap around to UINT_MAX when cast to an (unsigned int) making that above check true and so setting the backlog to /proc/sys/net/core/somaxconn.
So a first step could be to adjust the above to include Linux in setting the default backlog to -1. Maybe. FreeBSD seems to default to a value 128, on Linux since 5.4 it defaults to 4096 (previously 128)... this would also change it for the control socket...
Then again https://lore.kernel.org/netdev/20191030163620.140387-1-edumazet@google.com/ it sounds good!
Fun, fun, fun in the sun, sun sun...
Yes, 511 is outdated and limits server capabilities. Modern servers and new versions of PHP are becoming more and more powerful. It is time to adjust the backlog default value to maximize performance.
Linux version 5.15.153.1-microsoft-standard-WSL2 (root@941d701f84f1) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37)
Ok, good, if we change the default on Linux to use the OS's default, then you shouldn't actually need to do anything (unless you need to go higher than 4096...)
ok, i got it
RHEL 8 still ships with a 4.x series kernel.
Can we handle that by only setting it to -1 if the kernel version is >= 5.4? ...Or maybe it'd be easier to just unconditionally define NXT_LISTEN_BACKLOG as 4096 in all cases? If you need to go back down to 511 or 128, you can use the configuration option introduced in #1388 ?
I actually cover that here
Unless they are on an old Kernel, in which case it's worse, but then the
plan is to also make this configurable. This would effect RHEL 8, which
is based on 4.10, however they seem to set somaxconn to 2048, so that's
fine.
Also
Another advantage of using -1 is that we will automatically keep up to
date with the kernels default value.
Oh yep, you do. Fire away.