prometheus/blackbox_exporter

HTTP POST request ERROR: connect: cannot assign requested address

chichi13 opened this issue · 2 comments

chichi13 commented

Host operating system: output of uname -a

GKE (Google Kubernetes Engine) with container optimised OS.

blackbox_exporter version: output of blackbox_exporter --version

v0.24.0 in a kubernetes cluster

What is the blackbox.yml module config.

  http_post_2xx:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Content-Type: application/json
      body: '{}'

(As in the example.yml file at the repository root)

What logging output did you get from adding &debug=true to the probe URL?

Logs for the probe:
ts=2023-08-11T14:11:52.230021258Z caller=main.go:181 module=http_post_2xx target=prometheus.io level=info msg="Beginning probe" probe=http timeout_seconds=5
ts=2023-08-11T14:11:52.230148318Z caller=http.go:328 module=http_post_2xx target=prometheus.io level=info msg="Resolving target address" target=prometheus.io ip_protocol=ip6
ts=2023-08-11T14:11:52.252988787Z caller=http.go:328 module=http_post_2xx target=prometheus.io level=info msg="Resolved target address" target=prometheus.io ip=2606:4700:3030::ac43:c9f0
ts=2023-08-11T14:11:52.253116537Z caller=client.go:252 module=http_post_2xx target=prometheus.io level=info msg="Making HTTP request" url=http://[xxxx:xxxx:xxxx::xxxx:xxxx] host=prometheus.io
ts=2023-08-11T14:11:52.253301227Z caller=handler.go:120 module=http_post_2xx target=prometheus.io level=error msg="Error for HTTP request" err="Post \"http://[xxxx:xxxx:xxxx::xxxx:xxxx]\": dial tcp [xxxx:xxxx:xxxx::xxxx:xxxx]:80: connect: cannot assign requested address"
ts=2023-08-11T14:11:52.253331267Z caller=handler.go:120 module=http_post_2xx target=prometheus.io level=info msg="Response timings for roundtrip" roundtrip=0 start=2023-08-11T14:11:52.253225087Z dnsDone=2023-08-11T14:11:52.253225087Z connectDone=2023-08-11T14:11:52.253273247Z gotConn=0001-01-01T00:00:00Z responseStart=0001-01-01T00:00:00Z tlsStart=0001-01-01T00:00:00Z tlsDone=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2023-08-11T14:11:52.253358807Z caller=main.go:181 module=http_post_2xx target=prometheus.io level=error msg="Probe failed" duration_seconds=0.023285418

While my target URL is a domain name.

What did you do that produced an error?

What did you expect to see?

POST request working without connect: cannot assign requested address error.

What did you see instead?

Request failure because of a connect: cannot assign requested address error.

dswarbrick commented

blackbox_exporter defaults to using IPv6 to connect, if the target resolves to both IPv4 and IPv6 addresses. Does your host have IPv6 connectivity?

If yes, then the "cannot assign requested address" (which comes from the OS kernel) is likely an indicator that there are no spare / available source ports to use with that source IP.

chichi13 commented

It seems that you're right. I've put preferred_ip_protocol: "ip4" and it's working well now. I didn't know this config.

Thanks !