ateodorescu/home-assistant-addons

Unknown error occurred / 504 timeout - Error in IPMItool server when adding the IPMI Connector integration to Home Assistant

myklebosten opened this issue · 15 comments

myklebosten commented

Thanks for the integration and add-on!

I seem to be runing into a bug of some kind while adding the IPMI Connector to HA.

System details

I have installed both the HACS integration IPMI Connector and the Add-on IPMItool server.

My setup

Add-on: IPMItool server
IPMItool server

Add-on version: 1.1.25
You are running the latest version of this add-on.
System: Home Assistant OS 11.1 (amd64 / qemux86-64)
Home Assistant Core: 2023.11.2
Home Assistant Supervisor: 2023.11.3

All supplied logs originates from my HA OS running in a VM.
Also tested on dedicated hardware (HA Yellow) with same results.

IPMItool Server startup 

  s6-rc: info: service base-addon-banner successfully started
  s6-rc: info: service fix-attrs: starting
  s6-rc: info: service base-addon-log-level: starting
  s6-rc: info: service fix-attrs successfully started
  Log level is set to DEBUG
  s6-rc: info: service base-addon-log-level successfully started
  s6-rc: info: service legacy-cont-init: starting
  s6-rc: info: service legacy-cont-init successfully started
  s6-rc: info: service legacy-services: starting
  services-up: info: copying legacy longrun nginx (no readiness notification)
  services-up: info: copying legacy longrun php-fpm (no readiness notification)
  [16:07:31] INFO: Starting PHP-FPM...
  s6-rc: info: service legacy-services successfully started
  [16:07:32] INFO: Starting NGinx...

Problem

When adding the IPMI connector integration with known working connection details, I get an error:

(!) Unknown error occurred

IPMItool server Logs

2023/11/16 11:27:32 [error] 205#205: *17 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "my-iot-url.duckdns.org"

Ingress when using the URL from the log
URL: https:// my-iot-url.duckdns.org/api/hassio_ingress/3aZ8iC0mb6JKifA0UIsATgsPFg5Y5ww7pnUHdEcZX8w/?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword

504 Gateway Time-out
nginx

Further debuging

When adding the integration with with known not working connection details (like a bad password or host), i get a different error:

(!) Failed to connect

IPMItool server Logs

2023/11/16 11:27:32 [error] 205#205: *17 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "my-iot-url.duckdns.org"

Ingress when using the URL from the log
URL: https:// my-iot-url.duckdns.org/api/hassio_ingress/3aZ8iC0mb6JKifA0UIsATgsPFg5Y5ww7pnUHdEcZX8w/?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword

{"success":false,"message":"Wrong connection data provided!","debug":"Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info\u0022.\n
Unable to Get Channel Cipher Suites\nError: Unable to establish IPMI v2 \/ RMCP+ session\n\n

Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lan bmc info\u0022.\nAuthentication type NONE not supported\n
Error: Unable to establish LAN session\n
Error: Unable to establish IPMI v1.5 \/ RMCP session\n\n

Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I imb bmc info\u0022.\n
Error loading interface imb\n\n

Error occurred when running \u0022ipmitool -H 192.168.0.52-p 623 -U MyIPMIUsername -P MyPassword -I open bmc info\u0022.\n
Could not open device at \/dev\/ipmi0 or \/dev\/ipmi\/0 or \/dev\/ipmidev\/0: No such file or directory\n
"}

(I added new lines for every \n to make it more readable)

When runing the ipmitool command from Failed to connect with the correct data
from the Unknown error occurred, I get this:

Shell output: 

~$ ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info
Device ID                 : [Redacted]
Device Revision           : [Redacted]
Firmware Revision         : [Redacted]
IPMI Version              : 2.0
Manufacturer ID           : [Redacted]
Manufacturer Name         : [Redacted]
Product ID                : [Redacted]
Product Name              : [Redacted]
Device Available          : yes
Provides Device SDRs      : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    Chassis Device

(Tell me if you need any of the redacted info)

Appreciate if you have time to help me debug this.

I unfortunately don't have the skils to dig further by my self. Please tell me if there is anything I can do to assist :)

ateodorescu commented

Where did you run the working ipmitool command from? Was that inside the docker installation of the addon or just one of your systems?
Because you get a timeout I'm thinking that maybe the VM for HASS (including the docker container for the addon) has no access to the network of your server 192.168.0.52. Please look into this.

myklebosten commented

Thanks for the answer!

No, I ran ipmitool on my own linux host as I'm not that familiar with docker. But I read a tutorial and did connect to the docker host now. Here's the output:

root@304fecf0-ipmi-server:/$ ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info
Unable to Get Channel Cipher Suites
Device ID                 : [Redacted]
Device Revision           : [Redacted]
Firmware Revision         : [Redacted]
IPMI Version              : 2.0
Manufacturer ID           : [Redacted]
Manufacturer Name         : [Redacted]
Product ID                : [Redacted]
Product Name              : [Redacted]
Device Available          : yes
Provides Device SDRs      : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    Chassis Device

I also tested the integration and addon on dedicated hardware (Home Assistant Yellow) with same result. Wanted to be sure the VM environment wasn't the cause of the issue before posting. That info probably drowned in my way too long post yesterday.

ateodorescu commented

Please open in a browser this link: http://YOUR_HASS_SERVER_IP:9595/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info and let's see the result.

myklebosten commented

That one worked :)

{"success":true,"output":"Device ID                 : **\n
Device Revision           : *\n
Firmware Revision         : *.**\n
IPMI Version              : 2.0\n
Manufacturer ID           : **\n
Manufacturer Name         : *\n
Product ID                : **** (0x****)\n
Product Name              : Unknown (0x****)\n
Device Available          : yes\n
Provides Device SDRs      : no\n
Additional Device Support :\n
    Sensor Device\n
    SDR Repository Device\n
    SEL Device\n
    FRU Inventory Device\n
    Chassis Device\n
"}
myklebosten commented

By the way, still no success on adding the integration. And when using the URL from before (though last time I did it through ingress) I still get the 504 Gateway Time-out.

ateodorescu commented

Well, this is strange. Are you sure there is no typo in user/password/server_ip when typing them in the integration dialog?
I can't find a better reason for what is happening.

myklebosten commented

Agreed :)

Yes, I'm sure. I've typed it so many times now, I'm beginning to doubt myself. So this time I copied the info from the last known working command.

root@304fecf0-ipmi-server:/$ cat /var/log/php81/error.log
[16-Nov-2023 16:07:31] NOTICE: fpm is running, pid 156
[16-Nov-2023 16:07:31] NOTICE: ready to handle connections
root@304fecf0-ipmi-server:/$ cat /var/log/nginx/error.log

Any other logs I can look into?

ateodorescu commented

Any other logs I can look into?

None that I know of.

myklebosten commented

And as mentioned earlier: If I do type something wrong, I get a different error:

Integration: (!) Failed to connect
Add-on url: Produces output

If I have correct connection details, the integration produces the error:

Integration: (!) Unknown error occurred
Add-on url: Produces a 504 nginx error

myklebosten commented

I'll try to look into it a bit more when I have time later. Work (and/or lunch) is calling :)

myklebosten commented

I'm sorry, but I cant seem to find the problem. I've tried searching around a bit, and my guess is it's something to do with how nginx is communicating with php/fastcgi. If you want to continue, I'll do testing and stuff if needed. If not, you can close this issue :)

By the way: I would look into how to run this integration purely in ingress, with no external access. Shouldn't be too hard to do, just point your code to the internal docker host instead of an external host/IP on port 9595. That way, it won't be as easy to sniff the password over the network and get access to rebooting the server and what not.

ateodorescu commented

By the way: I would look into how to run this integration purely in ingress, with no external access. Shouldn't be too hard to do, just point your code to the internal docker host instead of an external host/IP on port 9595. That way, it won't be as easy to sniff the password over the network and get access to rebooting the server and what not.

I don't know how to do that. If you show me how then I could do it.

myklebosten commented

I'm not a HA dev, so I had to do some research for you on how to do this.

Integration and add-on today

You already have the ingress enabled in your add-on config, probably as per the docs here:
https://developers.home-assistant.io/docs/add-ons/presentation/#ingress
So I can enable the ipmi-tool in the sidebar of Home Assistant, and access it there (with urls and everything).

The add-on is installed with port 9595 enabled by default. This means the add-on webgui is accessible both via

  • ingress - with HA handling the login/access
  • externally on port 9595 - open access to anyone without login, and with no SSL/https support so the passwords can easily be sniffed with a simple network traffic analyzer (ipmi protocol might still send passwords in plain text though, I don't know).

Your integration (not add-on) needs the external port 9595 enabled, or it falls back to using python-ipmi. I have no integration coding nowledge, but as far as I can see line 21 in const.py is handling where to connect with the string IPMI_URL = "http://localhost". In the config flow of the integration, we can change the port.

Modifications needed

  • Point to add-on host instead of localhost
    I have little to no HA coding experience, but this looks promising:
    https://developers.home-assistant.io/docs/add-ons/communication/
    It states that to connect to the host, you can use the name/alias, which in your case shoud be 304fecf0_ipmi-server (hostname generated as per docs in the {REPO}_{SLUG} format, so it should stay constant as long as you don't change the slug ipmi-server).

  • Remove the port-option from config flow
    As this is now handled in config which is not user changable :D

  • Check your add-on configuration
    Docs: https://developers.home-assistant.io/docs/add-ons/presentation/#ingress

    • Ingress config
      Double check the config is correct.
    • Decide if you want to use the default port 8099
      or if you want to manually use another port. Just test what works, you'll figure it out :)
    • Disable external access on port 9595 by default

Final words

You told me to go to an url during debuging: http://YOUR_HASS_SERVER_IP:9595/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info.

You can do the same in ingress (if you didn't already know):
http[s]://YOUR_HA_URL/304fecf0_ipmi-server/ingress/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info

Hope this helps :)

  • 👍2
ateodorescu commented

Thanks! When I find the time I will look into that.

ateodorescu commented

@myklebosten try out the latest addon version 1.2.7 and see if that works for you. Thanks!