Unknown error occurred / 504 timeout - Error in IPMItool server when adding the IPMI Connector integration to Home Assistant
myklebosten opened this issue · 15 comments
Thanks for the integration and add-on!
I seem to be runing into a bug of some kind while adding the IPMI Connector to HA.
System details
I have installed both the HACS integration IPMI Connector and the Add-on IPMItool server.
My setup
Add-on: IPMItool server
IPMItool server
Add-on version: 1.1.25
You are running the latest version of this add-on.
System: Home Assistant OS 11.1 (amd64 / qemux86-64)
Home Assistant Core: 2023.11.2
Home Assistant Supervisor: 2023.11.3
All supplied logs originates from my HA OS running in a VM.
Also tested on dedicated hardware (HA Yellow) with same results.
IPMItool Server startup
s6-rc: info: service base-addon-banner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service base-addon-log-level: starting
s6-rc: info: service fix-attrs successfully started
Log level is set to DEBUG
s6-rc: info: service base-addon-log-level successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun nginx (no readiness notification)
services-up: info: copying legacy longrun php-fpm (no readiness notification)
[16:07:31] INFO: Starting PHP-FPM...
s6-rc: info: service legacy-services successfully started
[16:07:32] INFO: Starting NGinx...
Problem
When adding the IPMI connector integration with known working connection details, I get an error:
(!) Unknown error occurred
IPMItool server Logs
2023/11/16 11:27:32 [error] 205#205: *17 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "my-iot-url.duckdns.org"
Ingress when using the URL from the log
URL: https:// my-iot-url.duckdns.org/api/hassio_ingress/3aZ8iC0mb6JKifA0UIsATgsPFg5Y5ww7pnUHdEcZX8w/?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword
504 Gateway Time-out
nginx
Further debuging
When adding the integration with with known not working connection details (like a bad password or host), i get a different error:
(!) Failed to connect
IPMItool server Logs
2023/11/16 11:27:32 [error] 205#205: *17 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "my-iot-url.duckdns.org"
Ingress when using the URL from the log
URL: https:// my-iot-url.duckdns.org/api/hassio_ingress/3aZ8iC0mb6JKifA0UIsATgsPFg5Y5ww7pnUHdEcZX8w/?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword
{"success":false,"message":"Wrong connection data provided!","debug":"Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info\u0022.\n
Unable to Get Channel Cipher Suites\nError: Unable to establish IPMI v2 \/ RMCP+ session\n\n
Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lan bmc info\u0022.\nAuthentication type NONE not supported\n
Error: Unable to establish LAN session\n
Error: Unable to establish IPMI v1.5 \/ RMCP session\n\n
Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I imb bmc info\u0022.\n
Error loading interface imb\n\n
Error occurred when running \u0022ipmitool -H 192.168.0.52-p 623 -U MyIPMIUsername -P MyPassword -I open bmc info\u0022.\n
Could not open device at \/dev\/ipmi0 or \/dev\/ipmi\/0 or \/dev\/ipmidev\/0: No such file or directory\n
"}
(I added new lines for every \n to make it more readable)
When runing the ipmitool command from Failed to connect with the correct data
from the Unknown error occurred, I get this:
Shell output:
~$ ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info
Device ID : [Redacted]
Device Revision : [Redacted]
Firmware Revision : [Redacted]
IPMI Version : 2.0
Manufacturer ID : [Redacted]
Manufacturer Name : [Redacted]
Product ID : [Redacted]
Product Name : [Redacted]
Device Available : yes
Provides Device SDRs : no
Additional Device Support :
Sensor Device
SDR Repository Device
SEL Device
FRU Inventory Device
Chassis Device
(Tell me if you need any of the redacted info)
Appreciate if you have time to help me debug this.
I unfortunately don't have the skils to dig further by my self. Please tell me if there is anything I can do to assist :)
Where did you run the working ipmitool command from? Was that inside the docker installation of the addon or just one of your systems?
Because you get a timeout I'm thinking that maybe the VM for HASS (including the docker container for the addon) has no access to the network of your server 192.168.0.52. Please look into this.
Thanks for the answer!
No, I ran ipmitool on my own linux host as I'm not that familiar with docker. But I read a tutorial and did connect to the docker host now. Here's the output:
root@304fecf0-ipmi-server:/$ ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info
Unable to Get Channel Cipher Suites
Device ID : [Redacted]
Device Revision : [Redacted]
Firmware Revision : [Redacted]
IPMI Version : 2.0
Manufacturer ID : [Redacted]
Manufacturer Name : [Redacted]
Product ID : [Redacted]
Product Name : [Redacted]
Device Available : yes
Provides Device SDRs : no
Additional Device Support :
Sensor Device
SDR Repository Device
SEL Device
FRU Inventory Device
Chassis Device
I also tested the integration and addon on dedicated hardware (Home Assistant Yellow) with same result. Wanted to be sure the VM environment wasn't the cause of the issue before posting. That info probably drowned in my way too long post yesterday.
Please open in a browser this link: http://YOUR_HASS_SERVER_IP:9595/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info
and let's see the result.
That one worked :)
{"success":true,"output":"Device ID : **\n
Device Revision : *\n
Firmware Revision : *.**\n
IPMI Version : 2.0\n
Manufacturer ID : **\n
Manufacturer Name : *\n
Product ID : **** (0x****)\n
Product Name : Unknown (0x****)\n
Device Available : yes\n
Provides Device SDRs : no\n
Additional Device Support :\n
Sensor Device\n
SDR Repository Device\n
SEL Device\n
FRU Inventory Device\n
Chassis Device\n
"}
By the way, still no success on adding the integration. And when using the URL from before (though last time I did it through ingress) I still get the 504 Gateway Time-out.
Well, this is strange. Are you sure there is no typo in user/password/server_ip when typing them in the integration dialog?
I can't find a better reason for what is happening.
Agreed :)
Yes, I'm sure. I've typed it so many times now, I'm beginning to doubt myself. So this time I copied the info from the last known working command.
root@304fecf0-ipmi-server:/$ cat /var/log/php81/error.log
[16-Nov-2023 16:07:31] NOTICE: fpm is running, pid 156
[16-Nov-2023 16:07:31] NOTICE: ready to handle connections
root@304fecf0-ipmi-server:/$ cat /var/log/nginx/error.log
Any other logs I can look into?
Any other logs I can look into?
None that I know of.
And as mentioned earlier: If I do type something wrong, I get a different error:
Integration: (!) Failed to connect
Add-on url: Produces output
If I have correct connection details, the integration produces the error:
Integration: (!) Unknown error occurred
Add-on url: Produces a 504 nginx error
I'll try to look into it a bit more when I have time later. Work (and/or lunch) is calling :)
I'm sorry, but I cant seem to find the problem. I've tried searching around a bit, and my guess is it's something to do with how nginx is communicating with php/fastcgi. If you want to continue, I'll do testing and stuff if needed. If not, you can close this issue :)
By the way: I would look into how to run this integration purely in ingress, with no external access. Shouldn't be too hard to do, just point your code to the internal docker host instead of an external host/IP on port 9595. That way, it won't be as easy to sniff the password over the network and get access to rebooting the server and what not.
By the way: I would look into how to run this integration purely in ingress, with no external access. Shouldn't be too hard to do, just point your code to the internal docker host instead of an external host/IP on port 9595. That way, it won't be as easy to sniff the password over the network and get access to rebooting the server and what not.
I don't know how to do that. If you show me how then I could do it.
I'm not a HA dev, so I had to do some research for you on how to do this.
Integration and add-on today
You already have the ingress enabled in your add-on config, probably as per the docs here:
https://developers.home-assistant.io/docs/add-ons/presentation/#ingress
So I can enable the ipmi-tool in the sidebar of Home Assistant, and access it there (with urls and everything).
The add-on is installed with port 9595 enabled by default. This means the add-on webgui is accessible both via
- ingress - with HA handling the login/access
- externally on port 9595 - open access to anyone without login, and with no SSL/https support so the passwords can easily be sniffed with a simple network traffic analyzer (ipmi protocol might still send passwords in plain text though, I don't know).
Your integration (not add-on) needs the external port 9595 enabled, or it falls back to using python-ipmi. I have no integration coding nowledge, but as far as I can see line 21 in const.py is handling where to connect with the string IPMI_URL = "http://localhost"
. In the config flow of the integration, we can change the port.
Modifications needed
-
Point to add-on host instead of localhost
I have little to no HA coding experience, but this looks promising:
https://developers.home-assistant.io/docs/add-ons/communication/
It states that to connect to the host, you can use the name/alias, which in your case shoud be304fecf0_ipmi-server
(hostname generated as per docs in the{REPO}_{SLUG}
format, so it should stay constant as long as you don't change the slugipmi-server
). -
Remove the port-option from config flow
As this is now handled in config which is not user changable :D -
Check your add-on configuration
Docs: https://developers.home-assistant.io/docs/add-ons/presentation/#ingress- Ingress config
Double check the config is correct. - Decide if you want to use the default port
8099
or if you want to manually use another port. Just test what works, you'll figure it out :) - Disable external access on port 9595 by default
- Ingress config
Final words
You told me to go to an url during debuging: http://YOUR_HASS_SERVER_IP:9595/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info
.
You can do the same in ingress (if you didn't already know):
http[s]://YOUR_HA_URL/304fecf0_ipmi-server/ingress/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info
Hope this helps :)
Thanks! When I find the time I will look into that.
@myklebosten try out the latest addon version 1.2.7 and see if that works for you. Thanks!