prometheus/blackbox_exporter

Unable to resolve Static Targets

valyrian84 opened this issue · 1 comments

valyrian84 commented

None of my static targets are resolving -

Logs for the probe:
ts=2023-03-18T02:38:31.296004448Z caller=main.go:181 module=http_2xx target=https://example.com level=info msg="Beginning probe" probe=http timeout_seconds=4.5
ts=2023-03-18T02:38:31.297018411Z caller=http.go:327 module=http_2xx target=https://example.com level=info msg="Resolving target address" target=example.com ip_protocol=ip6
ts=2023-03-18T02:38:35.796829149Z caller=http.go:327 module=http_2xx target=https://example.com level=error msg="Resolution with IP protocol failed" target=example.com err="lookup example.com: i/o timeout"
ts=2023-03-18T02:38:35.797161556Z caller=handler.go:117 module=http_2xx target=https://example.com level=error msg="Error resolving address" err="lookup example.com: i/o timeout"
ts=2023-03-18T02:38:35.797305353Z caller=main.go:181 module=http_2xx target=https://example.com level=error msg="Probe failed" duration_seconds=4.50111009
-->

Host operating system: 5.15.84-v7l+ #1613 SMP Thu Jan 5 12:01:26 GMT 2023 armv7l GNU/Linux

blackbox_exporter version: output of blackbox_exporter --version

What is the blackbox.yml module config.

modules:
http_2xx:
prober: http
timeout: 5s
http:
method: GET
http_post_2xx:
prober: http
timeout: 5s
http:
method: POST
headers:
Content-Type: application/json
body: '{}'
tcp_connect:
prober: tcp
timeout: 5s
icmp:
prober: icmp
timeout: 5s

What is the prometheus.yml scrape config.

global:
scrape_interval: 5s
external_labels:
monitor: 'node'
scrape_configs:

  • job_name: 'prometheus'
    static_configs:
    • targets: ['192.168.x.x:9090'] ## IP Address of the localhost. Match the port to your container port
  • job_name: 'node-exporter'
    static_configs:
    • targets: ['192.168.x.x:9100'] ## IP Address of the localhost
  • job_name: 'blackbox-exporter'
    metrics_path: /probe
    params:
    module: [http_2xx]
    static_configs:

What logging output did you get from adding &debug=true to the probe URL?

Logs for the probe:
ts=2023-03-18T02:57:28.759639622Z caller=main.go:181 module=http_2xx target=prometheus.io level=info msg="Beginning probe" probe=http timeout_seconds=5
ts=2023-03-18T02:57:28.760779103Z caller=http.go:327 module=http_2xx target=prometheus.io level=info msg="Resolving target address" target=prometheus.io ip_protocol=ip6
ts=2023-03-18T02:57:33.759721637Z caller=http.go:327 module=http_2xx target=prometheus.io level=error msg="Resolution with IP protocol failed" target=prometheus.io err="lookup prometheus.io: i/o timeout"
ts=2023-03-18T02:57:33.759926785Z caller=handler.go:117 module=http_2xx target=prometheus.io level=error msg="Error resolving address" err="lookup prometheus.io: i/o timeout"
ts=2023-03-18T02:57:33.760047656Z caller=main.go:181 module=http_2xx target=prometheus.io level=error msg="Probe failed" duration_seconds=5.000168775

Metrics that would have been returned:

HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds

TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 4.998975664

HELP probe_duration_seconds Returns how long the probe took to complete in seconds

TYPE probe_duration_seconds gauge

probe_duration_seconds 5.000168775

HELP probe_failed_due_to_regex Indicates if probe failed due to regex

TYPE probe_failed_due_to_regex gauge

probe_failed_due_to_regex 0

HELP probe_http_content_length Length of http content response

TYPE probe_http_content_length gauge

probe_http_content_length 0

HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects

TYPE probe_http_duration_seconds gauge

probe_http_duration_seconds{phase="resolve"} 4.998975664

HELP probe_http_redirects The number of redirects

TYPE probe_http_redirects gauge

probe_http_redirects 0

HELP probe_http_ssl Indicates if SSL was used for the final redirect

TYPE probe_http_ssl gauge

probe_http_ssl 0

HELP probe_http_status_code Response HTTP status code

TYPE probe_http_status_code gauge

probe_http_status_code 0

HELP probe_http_uncompressed_body_length Length of uncompressed response body

TYPE probe_http_uncompressed_body_length gauge

probe_http_uncompressed_body_length 0

HELP probe_http_version Returns the version of HTTP of the probe response

TYPE probe_http_version gauge

probe_http_version 0

HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.

TYPE probe_ip_addr_hash gauge

probe_ip_addr_hash 0

HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6

TYPE probe_ip_protocol gauge

probe_ip_protocol 0

HELP probe_success Displays whether or not the probe was a success

TYPE probe_success gauge

probe_success 0

Module configuration:
prober: http
timeout: 5s
http:
ip_protocol_fallback: true
method: GET
follow_redirects: true
enable_http2: true
tcp:
ip_protocol_fallback: true
icmp:
ip_protocol_fallback: true
ttl: 64
dns:
ip_protocol_fallback: true
recursion_desired: true

dswarbrick commented

Is the blackbox_exporter host able to resolve those hostnames, e.g. using the host or dig tools?

PS: If you use enclose pasted content in triple backticks, it's a lot easier to read, and won't be interpreted as Markdown.