vozlt/nginx-module-vts

vhost_traffic_status_filter_by_set_key can be used to produce improperly formatted prometheus metrics

mpnowacki-reef opened this issue · 4 comments

mpnowacki-reef commented

For example, when adding filtering by user agent, if the filter label contains a " character, the produced prometheus metrics will look like:

nginx_vts_filter_requests_total{filter=""some-user-agent",filter_name="user-agent",direction="1xx"} 0

(notice the "" after filter=)

This breaks prometheus scraping. I guess this character needs to be escaped somehow.

u5surf commented

@mpnowacki-reef Hi, thanks for reporting. Can you share your nginx.conf and the request with the headers which reproduces this behavior.

mpnowacki-reef commented

Yes!

Here's the NGINX config:

server {
  listen      8000;
  add_header                Strict-Transport-Security "max-age=31536000" always;
  add_header                X-Content-Type-Options nosniff;
  add_header                X-XSS-Protection "1; mode=block";
  add_header                X-Frame-Options DENY;
  vhost_traffic_status_histogram_buckets 0.001 0.005 0.01 0.05 0.1 0.5 1 5 10 60 120;
  set $ws_or_http http;
  if ($http_upgrade) {
    set $ws_or_http ws;
  }
  vhost_traffic_status_filter_by_set_key ws_or_http $ws_or_http;
  set $user_agent_filter unknown;
  if ($http_user_agent) {
    set $user_agent_filter $http_user_agent;
  }
  vhost_traffic_status_filter_by_set_key user-agent $user_agent_filter;
  location / {
...
  }
}
server {
  listen      80;
  gzip off;
  access_log                /dev/stdout;
  error_log                 /dev/stderr info;
  client_max_body_size 100M;
  location /nginx-metrics {
    vhost_traffic_status_display;
    vhost_traffic_status_display_format prometheus;
  }
}

main nginx config contains vhost_traffic_status_zone in http {

a request that breaks scraping: curl address -A 'user-"agent'.

Following that request, prometheus fetches the metrics but refuses to decode them saying:

level=debug ts=2022-09-08T11:06:33.214Z caller=scrape.go:1274 component="scrape manager" scrape_pool=... target=.../nginx-metrics msg="Append failed" err="expected equal, got \"INVALID\""
u5surf commented

@mpnowacki-reef

Although I adopted your configuration onto latest(6fb2eb1), it looks fine, it can escape the double quote.

Have you tried the latest versions of this module?

# HELP nginx_vts_filter_request_seconds_total The request processing time in seconds counter
# TYPE nginx_vts_filter_request_seconds_total counter
# HELP nginx_vts_filter_request_seconds The average of request processing times in seconds
# TYPE nginx_vts_filter_request_seconds gauge
# HELP nginx_vts_filter_request_duration_seconds The histogram of request processing time
# TYPE nginx_vts_filter_request_duration_seconds histogram
# HELP nginx_vts_filter_cache_total The requests cache counter
# TYPE nginx_vts_filter_cache_total counter
nginx_vts_filter_bytes_total{filter="'user-\"agent'",filter_name="user-agent",direction="in"} 87
nginx_vts_filter_bytes_total{filter="'user-\"agent'",filter_name="user-agent",direction="out"} 2844
nginx_vts_filter_requests_total{filter="'user-\"agent'",filter_name="user-agent",code="1xx"} 0
nginx_vts_filter_requests_total{filter="'user-\"agent'",filter_name="user-agent",code="2xx"} 1
nginx_vts_filter_requests_total{filter="'user-\"agent'",filter_name="user-agent",code="3xx"} 0
nginx_vts_filter_requests_total{filter="'user-\"agent'",filter_name="user-agent",code="4xx"} 0
nginx_vts_filter_requests_total{filter="'user-\"agent'",filter_name="user-agent",code="5xx"} 0
nginx_vts_filter_request_seconds_total{filter="'user-\"agent'",filter_name="user-agent"} 0.000
nginx_vts_filter_request_seconds{filter="'user-\"agent'",filter_name="user-agent"} 0.000
nginx_vts_filter_cache_total{filter="'user-\"agent'",filter_name="user-agent",status="miss"} 0
nginx_vts_filter_cache_total{filter="'user-\"agent'",filter_name="user-agent",status="bypass"} 0
nginx_vts_filter_cache_total{filter="'user-\"agent'",filter_name="user-agent",status="expired"} 0
nginx_vts_filter_cache_total{filter="'user-\"agent'",filter_name="user-agent",status="stale"} 0
nginx_vts_filter_cache_total{filter="'user-\"agent'",filter_name="user-agent",status="updating"} 0
nginx_vts_filter_cache_total{filter="'user-\"agent'",filter_name="user-agent",status="revalidated"} 0
nginx_vts_filter_cache_total{filter="'user-\"agent'",filter_name="user-agent",status="hit"} 0
nginx_vts_filter_cache_total{filter="'user-\"agent'",filter_name="user-agent",status="scarce"} 0
u5surf commented

@mpnowacki-reef I could reproduce the issue by reverting this commit 9e814d0

Thus, we consider that the module which you use is a bit older than the latest.

Can you use the latest version(https://github.com/vozlt/nginx-module-vts/releases/tag/v0.2.1)?

If you cannot resolve the issue even though it updates the latest, can you tell us more details (e.g. nginx.conf, nginx -V)?