Howto get upstreams state
adrimt opened this issue · 5 comments
Hi,
the upstream state is always 'UP' even when the backend serer is down and the nginx return code is 502.
Did I miss something ?
vhost_traffic_status_zone;
geoip_country /usr/share/GeoIP/GeoIP.dat;
vhost_traffic_status_filter_by_set_key $geoip_country_code country::*;
vhost_traffic_status_zone shared:vhost_traffic_status:32m;
server {
listen 10.10.10.1:80;
server_name $hostname;
location /status {
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
access_log off;
}
}
server {
listen 10.10.10.10:80;
server_name webserver1;
location / {
proxy_pass http://192.168.10.10;
...
}
}
@adrimt Thanks reporting!
The other days, we had catch the same issue #258 (comment)
Then it could resolve the upstream zone.
Can you check this setting is existed on your nginx.conf (I could not find out upstream directive above config)
I change the config to define an "upstream" but it didn't behave differently.
The "check" directive is unavailable on my free nginx.
upstream web01-http {
zone backend 64k;
server 192.168.10.10:80;
}
server {
listen 10.10.10.10:80;
server_name webserver1;
location / {
proxy_pass http://web01-http;
...
}
@adrimt
This module seems to display whether the upstream state is down or not to watch the peer.down when display mode is html ( json mode is slightly different behavior on that #258 (comment))
nginx-module-vts/share/status.template.html
Line 685 in 5b3812e
Moreover nginx oss doesn't change peer->down anywhere below.
% rg "peer->down" ./src
./src/stream/ngx_stream_upstream_round_robin.c
461: if (peer->down) {
559: if (peer->down) {
./src/stream/ngx_stream_upstream_hash_module.c
235: if (peer->down) {
549: if (peer->down) {
./src/stream/ngx_stream_upstream_least_conn_module.c
142: if (peer->down) {
197: if (peer->down) {
./src/stream/ngx_stream_upstream_random_module.c
248: if (peer->down) {
350: if (peer->down) {
./src/http/ngx_http_upstream_round_robin.c
452: if (peer->down) {
550: if (peer->down) {
./src/http/modules/ngx_http_upstream_ip_hash_module.c
206: if (peer->down) {
./src/http/modules/ngx_http_upstream_random_module.c
248: if (peer->down) {
350: if (peer->down) {
./src/http/modules/ngx_http_upstream_least_conn_module.c
146: if (peer->down) {
201: if (peer->down) {
./src/http/modules/ngx_http_upstream_hash_module.c
236: if (peer->down) {
548: if (peer->down) {
We should match the behavior due to display mode both html and json.
By fixing this behavior, we can observe the upstream down to use nginx_upstream_check_module on workaround.
@adrimt
Hi, We end up checking the nginx upstream spec which the peer is single via nginx-dev ML. its answer is here.
https://mailman.nginx.org/pipermail/nginx-devel/2023-March/Q6MCXGQU56HDD7K2BWRYLCUEC7VBPUV4.html
As per nginx usage of upstreams with a single peer, such peers are
never considered unavailable, and to do this failures are never
counted (or, which is equivalent, immediately forgotten). If you are trying to show status of a peer as seen by nginx, then
"up" is perfectly correct for such peers.
Thus, we could not fix the view point of peer->fails with peers->single by any nginx upstream patch.
you should look into request results (and/or upstream states), and collect relevant stats yourself.
So we should embed other request based module to collect such stats, e.g. we can use nginx_upstream_check_module on workaround.
compile with pure nginx
$ git clone https://github.com/yaoweibin/nginx_upstream_check_module
$ cd /path/to/nginx # latest
$ patch -p1 < /path/to/nginx_upstream_check_module/check_1.20.1+patch
$ ./auto/configure --add-module=/path/to/nginx-module-vts --add-module=/path/to/nginx_upstream_check_module
$ make
$ sudo make install
nginx.conf
http {
...
vhost_traffic_status_zone;
upstream web {
zone backend 64k; # <- here
check interval=5000 rise=1 fall=3 timeout=4000;
server 127.0.0.1:8080;
}
server {
listen 8000;
location / {
proxy_pass http://web;
}
}
server {
listen 80;
location /status/ {
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
}
}
The upstream peer is up
otherwise
Thank you for clarifying this issue.