prometheus/blackbox_exporter

Curl and blackbox http_2xx are not matching total time, showing hugely different figures

Closed this issue · 1 comments

varunpalekar commented

Host operating system: output of uname -a

Linux 1a011975b554 6.11.5-orbstack-00280-g96d99c92a42b #51 SMP Sun Nov 3 08:07:37 UTC 2024 aarch64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter --version

blackbox_exporter, version 0.25.0 (branch: HEAD, revision: ef3ff4fef195333fb8ee0039fb487b2f5007908f)
  build user:       root@9b6755681f2e
  build date:       20240409-12:52:48
  go version:       go1.22.2
  platform:         linux/arm64
  tags:             unknown

What is the blackbox.yml module config.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      method: GET
  http_307:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes:
        - 307

What is the prometheus.yml scrape config.

NA

What logging output did you get from adding &debug=true to the probe URL?

NA

What did you do that produced an error?

  1. Start blackbox using following docker compose
services:
  blackbox:
    image: quay.io/prometheus/blackbox-exporter:v0.25.0
    ports:
    - 9115:9115
    command:
    - --config.file=/etc/blackbox/blackbox.yml
    volumes:
    - ./blackbox.yml:/etc/blackbox/blackbox.yml

blackbox.yml file

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      method: GET
  http_307:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes:
        - 307
  1. Get data from curl
# docker run -it curlimages/curl -o /dev/null -s -w "dns:%{time_namelookup} , redir:%{time_redirect}, tcp:%{time_connect}, total:%{time_total}s" https://jsonplaceholder.typicode.com/todos/1

dns:0.001408 , redir:0.000000, tcp:0.088904, total:0.373054s%
  1. Get same data from blackbox probe
> curl "http://localhost:9115/probe?target=http://jsonplaceholder.typicode.com/todos/1&module=http_2xx"
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 0.004335141
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.004864767
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HELP probe_http_content_length Length of http content response
# TYPE probe_http_content_length gauge
probe_http_content_length 0
# HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects
# TYPE probe_http_duration_seconds gauge
probe_http_duration_seconds{phase="connect"} 0
probe_http_duration_seconds{phase="processing"} 0
probe_http_duration_seconds{phase="resolve"} 0.004335141
probe_http_duration_seconds{phase="tls"} 0
probe_http_duration_seconds{phase="transfer"} 0
# HELP probe_http_redirects The number of redirects
# TYPE probe_http_redirects gauge
probe_http_redirects 0
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 0
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 0
# HELP probe_http_uncompressed_body_length Length of uncompressed response body
# TYPE probe_http_uncompressed_body_length gauge
probe_http_uncompressed_body_length 0
# HELP probe_http_version Returns the version of HTTP of the probe response
# TYPE probe_http_version gauge
probe_http_version 0
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 4.264684175e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 6
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 0

What did you expect to see?

probe_duration_seconds must be near curl total time

What did you see instead?

probe_duration_seconds: 0.004864767
curl total time: 0.373054

I also checked if curl is doing something wrong but verify this data using (otel httpcheck)[https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/httpcheckreceiver/documentation.md#httpcheckduration] and it matching with curl value i.e., 120ms

varunpalekar commented

I guess found the problem, as status code is 0, that means something is not working.
and I found out that default it works on IPV6 which is not enabled in my network. So when I switch to ipv4 it start working properly, closing it

  • 👍1