vsphere temporary files
david-ungurean opened this issue · 5 comments
These files should not be there. Could you please share:
icingacli version
journalctl -u icinga-vspheredb.service --since 09:00 --until 12:00
(please anonymize the output, in case it contains sensitive data)
Here is the content:
icingacli version
journalctl ....
- Logs begin at Tue 2023-01-31 08:55:41 UTC, end at Tue 2023-01-31 12:39:58 UTC. --
Jan 31 09:00:00 icinga2-server icinga-vspheredb[25054]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Watchdog timeout (limit 10s)!
Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Killing process 25054 (icingacli) with signal SIGABRT.
Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Killing process 25055 (icingacli) with signal SIGABRT.
Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Main process exited, code=killed, status=6/ABRT
Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Failed with result 'watchdog'.
Jan 31 09:08:30 icinga2-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
Jan 31 09:08:30 icinga2-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1211.
Jan 31 09:08:30 icinga2-server systemd[1]: Stopped Icinga vSphereDB Daemon.
Jan 31 09:08:30 icinga2-server systemd[1]: Starting Icinga vSphereDB Daemon...
Jan 31 09:08:32 icinga2-server systemd[1]: Started Icinga vSphereDB Daemon.
Jan 31 09:08:32 icinga2-server icinga-vspheredb[25150]: [configwatch] DB configuration loaded
Jan 31 09:08:32 icinga2-server icinga-vspheredb[25150]: [db] sending DB config to child process
Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [db] Running DB cleanup (this could take some time)
Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [db] DB has been cleaned up
Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [localdb] ready
Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [api] launching server 2: vCenterId=1: https://user@datacenter
Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Logged out
Jan 31 09:08:34 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Cookies changed, storing new ones
Jan 31 09:08:38 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:40 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:42 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:44 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:46 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:49 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:53 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:55 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:57 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:08:59 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:09:01 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:09:03 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:09:05 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:09:07 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:10:05 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:18:44 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:18:51 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:18:56 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:18:58 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:01 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:03 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:05 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:07 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:09 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:11 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:13 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:15 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:17 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:19 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:21 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:23 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:25 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:27 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:29 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:31 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:33 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:34 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'VM Quick Stats' is already running, skipping
Jan 31 09:19:35 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:37 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:39 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:41 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:43 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:45 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:49 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:51 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:53 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:56 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:58 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:19:59 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:28:32 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Watchdog timeout (limit 10s)!
Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25150 (icingacli) with signal SIGABRT.
Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25151 (icingacli) with signal SIGABRT.
Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Main process exited, code=killed, status=6/ABRT
Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Failed with result 'watchdog'.
Jan 31 09:29:13 icinga-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
Jan 31 09:29:13 icinga-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1212.
Jan 31 09:29:13 icinga-server systemd[1]: Stopped Icinga vSphereDB Daemon.
Jan 31 09:29:13 icinga-server systemd[1]: Starting Icinga vSphereDB Daemon...
Jan 31 09:29:14 icinga-server systemd[1]: Started Icinga vSphereDB Daemon.
Jan 31 09:29:14 icinga-server icinga-vspheredb[25268]: [configwatch] DB configuration loaded
Jan 31 09:29:14 icinga-server icinga-vspheredb[25268]: [db] sending DB config to child process
Jan 31 09:29:15 icinga-server icinga-vspheredb[25268]: [db] Running DB cleanup (this could take some time)
Jan 31 09:29:15 icinga-server icinga-vspheredb[25268]: [db] DB has been cleaned up
Jan 31 09:29:15 icinga-server icinga-vspheredb[25268]: [localdb] ready
Jan 31 09:29:15 icinga-server icinga-vspheredb[25268]: [api] launching server 2: vCenterId=1: https://user@aa.bb.cc.dd
Jan 31 09:29:16 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Logged out
Jan 31 09:29:16 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
Jan 31 09:29:20 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:22 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:24 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:27 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:28 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:30 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:34 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:36 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:38 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:40 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:42 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:44 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:29:46 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:30:47 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Watchdog timeout (limit 10s)!
Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25268 (icingacli) with signal SIGABRT.
Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25269 (icingacli) with signal SIGABRT.
Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Main process exited, code=killed, status=6/ABRT
Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Failed with result 'watchdog'.
Jan 31 09:39:55 icinga-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
Jan 31 09:39:55 icinga-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1213.
Jan 31 09:39:55 icinga-server systemd[1]: Stopped Icinga vSphereDB Daemon.
Jan 31 09:39:56 icinga-server systemd[1]: Starting Icinga vSphereDB Daemon...
Jan 31 09:39:59 icinga-server systemd[1]: Started Icinga vSphereDB Daemon.
Jan 31 09:39:59 icinga-server icinga-vspheredb[25400]: [configwatch] DB configuration loaded
Jan 31 09:39:59 icinga-server icinga-vspheredb[25400]: [db] sending DB config to child process
Jan 31 09:40:00 icinga-server icinga-vspheredb[25400]: [db] Running DB cleanup (this could take some time)
Jan 31 09:40:00 icinga-server icinga-vspheredb[25400]: [db] DB has been cleaned up
Jan 31 09:40:00 icinga-server icinga-vspheredb[25400]: [localdb] ready
Jan 31 09:40:00 icinga-server icinga-vspheredb[25400]: [api] launching server 2: vCenterId=1: https://user@aa.bb.cc.dd
Jan 31 09:40:01 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Logged out
Jan 31 09:40:02 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
Jan 31 09:40:08 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:10 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:12 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:14 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:18 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:20 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:22 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:24 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:26 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:28 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:40:30 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:41:32 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:49:03 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:50:08 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:50:15 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:50:19 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:50:21 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:50:23 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:50:25 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:50:27 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:50:29 icinga-server icingacli[25400]: ERROR: ErrorException in /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Daemon/RemoteApi.php:103 with message: shell_exec(): Unable to execute 'id -G 33'
Jan 31 09:50:30 icinga-server systemd[1]: icinga-vspheredb.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 09:50:30 icinga-server systemd[1]: icinga-vspheredb.service: Failed with result 'exit-code'.
Jan 31 09:51:00 icinga-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
Jan 31 09:51:00 icinga-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1214.
Jan 31 09:51:00 icinga-server systemd[1]: Stopped Icinga vSphereDB Daemon.
Jan 31 09:51:00 icinga-server systemd[1]: Starting Icinga vSphereDB Daemon...
Jan 31 09:51:02 icinga-server systemd[1]: Started Icinga vSphereDB Daemon.
Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [configwatch] DB configuration loaded
Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [db] sending DB config to child process
Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [db] Running DB cleanup (this could take some time)
Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [db] DB has been cleaned up
Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [localdb] ready
Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [api] launching server 2: vCenterId=1: https://user@aa.bb.cc.dd
Jan 31 09:51:03 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Logged out
Jan 31 09:51:03 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
Jan 31 09:51:10 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:12 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:14 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:17 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:19 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:23 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:25 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:27 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:29 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:31 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:33 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:35 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:37 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:51:39 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:52:35 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 09:54:05 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:12 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:17 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:19 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:21 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:23 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:25 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:27 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:29 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:31 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:33 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:35 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:37 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:39 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:41 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:43 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:45 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:47 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:49 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:51 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:53 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:55 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:01:57 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:01 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:03 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:03 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'VM Quick Stats' is already running, skipping
Jan 31 10:02:05 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:07 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:09 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:11 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:13 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:15 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:17 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:02:19 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:03:18 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:06:18 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:09:19 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Watchdog timeout (limit 10s)!
Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25495 (icingacli) with signal SIGABRT.
Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25498 (icingacli) with signal SIGABRT.
Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Main process exited, code=killed, status=6/ABRT
Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Failed with result 'watchdog'.
Jan 31 10:11:43 icinga-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
Jan 31 10:11:43 icinga-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1215.
Jan 31 10:11:43 icinga-server systemd[1]: Stopped Icinga vSphereDB Daemon.
Jan 31 10:11:43 icinga-server systemd[1]: Starting Icinga vSphereDB Daemon...
Jan 31 10:11:45 icinga-server systemd[1]: Started Icinga vSphereDB Daemon.
Jan 31 10:11:45 icinga-server icinga-vspheredb[25645]: [configwatch] DB configuration loaded
Jan 31 10:11:45 icinga-server icinga-vspheredb[25645]: [db] sending DB config to child process
Jan 31 10:11:46 icinga-server icinga-vspheredb[25645]: [db] Running DB cleanup (this could take some time)
Jan 31 10:11:46 icinga-server icinga-vspheredb[25645]: [db] DB has been cleaned up
Jan 31 10:11:46 icinga-server icinga-vspheredb[25645]: [localdb] ready
Jan 31 10:11:46 icinga-server icinga-vspheredb[25645]: [api] launching server 2: vCenterId=1: https://user@aa.bb.cc.dd
Jan 31 10:11:47 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Logged out
Jan 31 10:11:47 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
Jan 31 10:11:52 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:11:54 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:11:56 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:11:58 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:12:02 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:12:04 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:12:06 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 10:12:08 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Thank you!
Your daemon is dying again and again, being killed by systemd because of watchdog timeouts. Is your database running normally? As you mentioned a full disk, has the disk been full? Anything special about this setup? Very large vCenter, slow DB (containerized, object store, cloud service)?
Please restart the daemon, wait 10 seconds - then run systemctl status icinga-vspheredb.service
, and share the output. Also, a SHOW PROCESSLIST;
for your DB would be great, also executed about 10 seconds in. Please share some rough details (how many processes, are there any blocking ones or other special states) - and especially the processes related to vSphereDB.
I think I found the problem.
I run free -m and notice that the server is not having enough memory (he only had 1 GB of RAM), so I added 3 GB of RAM, and now seems to be ok.
I suppose this will solve the problem.
Anyhow, I deleted the tmp files and restarted all the services.
The results after reboot:
Jan 31 13:22:09 icinga-server icinga-vspheredb[598]: [db] DB has been cleaned up
Jan 31 13:22:09 icinga-server icinga-vspheredb[598]: [localdb] ready
Jan 31 13:22:09 icinga-server icinga-vspheredb[598]: [api] launching server 2: vCenterId=1: https://usero@aa.bb.cc.dd
Jan 31 13:22:12 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
Jan 31 13:22:17 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:22:20 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:22:24 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:22:28 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:22:30 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:23:42 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Thank you!
David
You're welcome!