Error creating redfish client app=redfish_exporter
Closed this issue · 19 comments
``Hi. I am stuck.
Goal = Trying to use the redfish_exporter to load sensor data from BMC via redfish and track that sensor data in Prometheus.
Setup =
- Prometheus service running on CentOS host with IP 10.219.111.125.
- This is a CentOS Linux release 8.4.2105
- That host's BMC IP is 10.219.111.126. This is where i want to send redfish commands to load up sensor data.
- The redfish_exporter's config yml file lists 10.219.111.126 , and includes the username and password to enter the BMC.
- I am able to point the host machine's browser to 10.219.111.125:9090 and it loads Prometheus
- I am able to point the host machine's browser to 10.219.111.125:9610 and it shows the redfish_exporter UI
- In that redfish_exporter UI , i can enter 10.219.111.126 and it loads up data, but i noticed it say "redfish_up 0"
Problem = I keep seeing the following error
error creating redfish client app=redfish_exporter error=Get "https://10.219.111.125/redfish/v1": proxyconnect tcp: tls: first record does not look like a TLS handshake target=10.219.111.125
Prometheus is running as a service. In /etc/prometheus/prometheus.yml i have the following section...
- job_name: 'redfish-exporter'
# metrics_path defaults to '/metrics'
metrics_path: /redfish
# scheme defaults to 'http'.
static_configs:
- targets:
- 10.219.111.125
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9610 ### the address of the redfish-exporter address
# (optional) when using group config add this to have group=my_group_name
# target_label: __param_group
# replacement: my_group_name
What am i doing wrong?
Thank you for your help!
What do you see if you point your browser at https://10.219.111.125/redfish/v1
From inside the host itself, if i point to https://10.219.111.125/redfish/v1 i get to connect "unable to connect". Same if i just say https://localhost/redfish/v1. Same if i point to that from a desktop that is able to reach http://10.219.111.125:9090 and see Prometheus.
Are IPs that i entered in my prometheus.yml and redfish_exporter config.yml correct?
For prometheus I set the "target" to .125, since that is the host OS in which the redfish_exporter is running, but for redfish_exporter I set the "host" to .126 since that is the BMC from which i need to pull the sensor data. Is that correct?
Something is not configured right if the BMC address is .126. From that log, it is trying to access the redfish API (via the BMC) at the .125 address. Try setting "target" to .126.
You should also be able to verify redfish is working by going to https://10.219.111.126/redfish/v1 from inside the host. That might be a good validation check just to make sure the service is operating correctly, though it does look like the main issue is just that it is going to the wrong IP when trying to access it.
thank you.
From inside the host, i am able to browse to 10.219.111.126/redfish/v1, but that should work regardless of prometheus and regardless of the redfish_exporter because i'm hitting the BMC directly that way. So it is expected to work and it is.
I updated the target in prometheus.yml to .126 and restarted the prometheus server. I also restarted redfish_exporter. I still see the following in the logs.
i just set no_proxy and ftp_proxy and socks_proxy. It seems to be working now. Can you please confirm?
When i browse to the redfish_exporter on the host and enter .126 as follows:
I see the redfish_exporter logs give me the following...
I don't see the TLS error. So maybe its working now? How can i check for sure?
What query do i type in Prometheus to see metrics of what's being scrapped?
why should the "target" in prometheus.yml point to .126? That's the BMC address. Shouldn't it point to .125 (which is what i originally had), since .125 is where the redfish_exporter is running. The redfish_exporter would load telemetry from the BMC at .126.
I don't get how prometheus and redfish_exporter are expected to be connected after i applied your recommended change of setting the target to .126
Sorry, you're probably right. Maybe someone else can jump in here with some suggestions. Just wanted to check since that error message seemed to indicate it was trying to connect to the host IP to access the redfish API rather than the BMC IP address.
Thanks. here is the current state i'm in...
The config.yml for redfish_exporter includes 10.219.111.126 (BMC) in the "hosts:" section. That's showing proper scraping as per screenshot i shared earlier
I updated prometheus.yml to point to .125 (host ip where redfish_exporter is running). Then restarted prometheus.
From the host's browser (i.e. on .125), i ran http://10.219.111.125/redfish?target=10.219.111.126 and it takes a while and loads. If i scroll to the bottom, i see "redfish_up 1". see below. So that looks like its working.
However, prometheus is seeing 0 for redfish_up. What am i doing wrong here?
Thanks
Hi,
I also felt strange, why promethus didn't call the real ip address of BMC, can you please post following things.
- redfish_exporter config
- prometehus config regarding redfish job
- both redfish_exporter and prometheus logs
take this redfish config for example
- job_name: 'redfish-exporter'
# metrics_path defaults to '/metrics'
metrics_path: /redfish
# scheme defaults to 'http'.
static_configs:
- targets:
- 10.219.111.125
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9610 ### the address of the redfish-exporter address
# (optional) when using group config add this to have group=my_group_name
# target_label: __param_group
- target_label: __address__
replacement: localhost:9610 ### the address of the redfish-exporter address
as the comments pointed out, here is the ip address of redfish-export, hence the localhost should be replaced to the IP address of redfish-exporter
static_configs:
- targets:
- 10.219.111.125
here is ip address of your server, which you are willing to monitor, so here you shouldn't add the ip address of redfish, but modified to bmc ip address
hi. Here are the 3 files in question. I have redacted some items and replaced with XXX
Thank you for your help.
-
Redfish exporter config
-
Prometheus config
-
Logs
Redfish_exporter log looks good..
Prometheus exporter log looks good
Notice that i can run this command no problem from the browser http://10.219.111.125:9610/redfish?target=10.219.111.126&group= and i get redfish_up 1
Also, in prometheus, i see the following...
However, redfish_up always shows a 0 as follows:
As a reminder.. 10.219.111.125 is the OS, and 10.219.111.126 is the BMC... both are the same node.
see my last comments to configure your prometheus， read the comments and the readme more carefully before you configure both prometheus and redfish.
here you misunderstand the addresses, please check prometheus config once more.
Regarding your last comment to configure prometheus, you said "here is ip address of your server, which you are willing to monitor, so here you shouldn't add the ip address of redfish, but modified to bmc ip address", but as I mentioned before, i tried both 125 and 126.
I went ahead again and changed this back to .126, restarted the prometheus service and restarted the redfish exporter just to be safe. I am still getting redfish_up = 0 in prometheus but I get 1 when i invoke the exporter through the web. What's worse is now the redfish_exporter is showing errors. See below.
So far, i have provided you with all the details and i tried multiple settings, but the exporter and prometheus are not connecting together. Please advise.
your logs and screenshot both show you still configure the monitored target to 10.219.111.125 which is the address of redfish, that is the problem, it should be 10.219.111.126
there must be wrong config at your promethus side, as I used this in my environment for years, no such issue
you can refer to my config here
and the content of tgroups/redfish-oob-targets.yml
You said "your logs and screenshot both show you still configure the monitored target to 10.219.111.125 which is the address of redfish". 125 is not the address of redfish. 125 is the address of the unix box on which the redfish_exporter is running. 126 is the address of the BMC which is redfish-capable.
Earlier in this thread, i had the targets section point to 125. The redfish_exporter was not showing errors, but Prometheus was not able to see the exporter. Then you told me to change the targets section to point to 126, and i did. That's when the redfish_exporter started to show errors and prometheus still could not see the exporter.
Please tell me exactly how you recommend that configure this. Again, 125 is unix on which redfish_exporter is running and 126 is redfish-capable BMC.
Thanks.
I said you should configure your BMC IP address in the stactic-configs section which is 126.
if your can post your screenshot regarding the promethus status-->targets page
I have to say, there are a bunch of users used this tools, but none of them have such configuration problem, so firstly please read more carefully about the promethus documents
I will close this issue, as it is really your configuration error and may not fully understand prometheus---> redfish-exporter-->monitor target model.
I resolved the issue. It was not my configuration. The Prometheus service was in a zombie state. I had to clean up the /var/lib/prometheus/lock file