UDMB sporadic data after update to 1.3

Question

UDMB sporadic data after update to 1.3

pax0707 opened this issue 2 years ago · 12 comments

Anything I can check here?
AP and switch are reporting fine but UDN (base, not pro one) does not play nice.

Answer 1 · 2022-12-16T18:10:11.000Z

I just updated one to 1.3 as well.. it's playing nice here so far.

Anything in the zabbix server logs? when mca-dump-short goes not supported on the controller host like above, what's the error message?

[edit] mine is a pro, not a base so that could be a difference

Answer 2 · 2022-12-20T19:58:37.000Z

838333:20221220:202552.874 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838327:20221220:202652.700 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838334:20221220:202752.695 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838323:20221220:202852.812 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838324:20221220:202952.658 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838334:20221220:203052.505 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838323:20221220:203152.607 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838311:20221220:203253.214 item "UniFi Controller:mca-dump-short.sh["-d","{HOST.CONN}", "-u", "root", "-i", "{$UNIFI_SSH_PRIV_KEY_PATH}", "-t", "UDMP", "-p", "{$UNIFI_SSHPASS_PASSWORD_PATH}", "-o", "{$UNIFI_CHECK_TIMEOUT}" ]" became supported
838323:20221220:203352.930 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838311:20221220:203353.103 item "UniFi Controller:mca-dump-short.sh["-d","{HOST.CONN}", "-u", "root", "-i", "{$UNIFI_SSH_PRIV_KEY_PATH}", "-t", "UDMP", "-p", "{$UNIFI_SSHPASS_PASSWORD_PATH}", "-o", "{$UNIFI_CHECK_TIMEOUT}" ]" became not supported: Timeout while executing a shell script.
838327:20221220:203452.873 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838327:20221220:203552.767 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838323:20221220:203652.702 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838333:20221220:203752.631 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838324:20221220:203852.546 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838327:20221220:203952.481 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838323:20221220:204052.424 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838327:20221220:204152.339 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838334:20221220:204252.363 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.
838324:20221220:204352.167 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'": Timeout while executing a shell script.

When I run "sudo -u zabbix ./mca-dump-short.sh -i ./zb_id_rsa -u root -v -d 172.16.1.1 -t UDMP" it runs., About 2-3 seconds to execute.
Tried raising timeout to 30, still timeouts.

Answer 3 · 2022-12-21T22:48:11.000Z

mm.. have you tried running exactly the same command:

/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '172.16.1.1' '-u' 'root' '-i' '/usr/lib/zabbix/externalscripts/zb_id_rsa' '-t' 'UDMP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '30'

?

anything in /tmp/mcaDumpShort.err on the zabbix server?

Answer 4 · 2022-12-21T23:29:25.000Z

Output:
https://gist.github.com/pax0707/701ae085daf2b61a99f23d6fd60b2cfb

tmp dir is full of empty mca*.err files.

-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:18 mca-14284.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:17 mca-18696.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:20 mca-19170.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:25 mca-22353.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:28 mca-22716.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:23 mca-23358.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:26 mca-27412.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:22 mca-30038.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:15 mca-30896.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:16 mca-31093.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:21 mca-4403.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:24 mca-4862.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:19 mca-5992.err
-rw-rw-r--  1 zabbix zabbix    0 Dec 22 00:27 mca-8796.err

Answer 5 · 2022-12-21T23:42:20.000Z

pi@rpi3:/tmp $ cat mcaDumpShort.err
Thu 22 Dec 2022 12:35:49 AM CET 172.16.1.1

Thu 22 Dec 2022 12:35:49 AM CET 172.16.1.89

Thu 22 Dec 2022 12:36:13 AM CET 172.16.1.97

Thu 22 Dec 2022 12:36:17 AM CET 172.16.1.1

Thu 22 Dec 2022 12:36:19 AM CET 172.16.1.1

Thu 22 Dec 2022 12:36:35 AM CET 172.16.1.1

Thu 22 Dec 2022 12:36:41 AM CET 172.16.1.1

Thu 22 Dec 2022 12:36:49 AM CET 172.16.1.1

Thu 22 Dec 2022 12:36:50 AM CET 172.16.1.89

Thu 22 Dec 2022 12:37:13 AM CET 172.16.1.97

Thu 22 Dec 2022 12:37:48 AM CET 172.16.1.1

Thu 22 Dec 2022 12:37:50 AM CET 172.16.1.89

Thu 22 Dec 2022 12:37:57 AM CET 172.16.1.1

Thu 22 Dec 2022 12:37:58 AM CET 172.16.1.1

Thu 22 Dec 2022 12:38:13 AM CET 172.16.1.97

Thu 22 Dec 2022 12:38:31 AM CET 172.16.1.1

Thu 22 Dec 2022 12:38:49 AM CET 172.16.1.1

Thu 22 Dec 2022 12:38:49 AM CET 172.16.1.89

Thu 22 Dec 2022 12:39:07 AM CET 172.16.1.1

Thu 22 Dec 2022 12:39:13 AM CET 172.16.1.97

pi@rpi3:/tmp $

Answer 6 · 2022-12-22T00:33:32.000Z

This is Fed up...
Restarted UDM and everything is working....

Answer 7 · 2022-12-22T16:43:58.000Z

Didn't last....

"Timeout while executing a shell script."

It's very annoying to troubleshoot since it works for a bit, starts to die, and then mostly doesn't work.

Answer 8 · 2022-12-22T16:47:45.000Z

What is your overall TIMEOUT value set in the Zabbix conf?Sent from my iPadOn Dec 22, 2022, at 8:44 AM, pax0707 ***@***.***> wrote: Didn't last.... "Timeout while executing a shell script." —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

Answer 9 · 2023-01-11T23:47:57.000Z

hi @pax0707 - could I get you to try with an updated mcaDumpShort.sh and templates? the situation around timeouts has been improved and there also should be more info in /tmp/mcaDumpShort.err on the zabbixXerver

Answer 10 · 2023-01-12T20:19:12.000Z

pi@rpi3:~ $ cat /tmp/mcaDumpShort.err 
----------------------------------
Thu 12 Jan 2023 09:17:15 PM CET 172.16.1.97
   { "at":"21:17:15", "r":"please install expect to run SWITCH_DISCOVERY", "device":"172.16.1.97", "mcaDumpError":"Error" }\n

Gonna deal with this while at it.

Answer 11 · 2023-01-12T20:53:07.000Z

Working so far, monitoring...

6 hours later and still ok.
Gonna close this tomorrow if it continues to be.

Answer 12 · 2023-01-13T21:39:00.000Z

Working fine.