Icinga/icingaweb2-module-vspheredb

vsphere temporary files

david-ungurean opened this issue · 5 comments

david-ungurean commented

Hello,

Icingaweb2-module-sphere is creating XML files and storing them in /tmp folder until the disk is full.

image

Is there any configuration that I can make to auto-clean them?
For the moment I created a script which is cleaning tmp folder, but I think there should be another solution.

Thank you,

David

Thomas-Gelf commented

These files should not be there. Could you please share:

  • icingacli version
  • journalctl -u icinga-vspheredb.service --since 09:00 --until 12:00 (please anonymize the output, in case it contains sensitive data)
david-ungurean commented

Here is the content:

icingacli version

image

journalctl ....

  • Logs begin at Tue 2023-01-31 08:55:41 UTC, end at Tue 2023-01-31 12:39:58 UTC. --
    Jan 31 09:00:00 icinga2-server icinga-vspheredb[25054]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Watchdog timeout (limit 10s)!
    Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Killing process 25054 (icingacli) with signal SIGABRT.
    Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Killing process 25055 (icingacli) with signal SIGABRT.
    Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Main process exited, code=killed, status=6/ABRT
    Jan 31 09:07:59 icinga2-server systemd[1]: icinga-vspheredb.service: Failed with result 'watchdog'.
    Jan 31 09:08:30 icinga2-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
    Jan 31 09:08:30 icinga2-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1211.
    Jan 31 09:08:30 icinga2-server systemd[1]: Stopped Icinga vSphereDB Daemon.
    Jan 31 09:08:30 icinga2-server systemd[1]: Starting Icinga vSphereDB Daemon...
    Jan 31 09:08:32 icinga2-server systemd[1]: Started Icinga vSphereDB Daemon.
    Jan 31 09:08:32 icinga2-server icinga-vspheredb[25150]: [configwatch] DB configuration loaded
    Jan 31 09:08:32 icinga2-server icinga-vspheredb[25150]: [db] sending DB config to child process
    Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [db] Running DB cleanup (this could take some time)
    Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [db] DB has been cleaned up
    Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [localdb] ready
    Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [api] launching server 2: vCenterId=1: https://user@datacenter
    Jan 31 09:08:33 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Logged out
    Jan 31 09:08:34 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Cookies changed, storing new ones
    Jan 31 09:08:38 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:40 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:42 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:44 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:46 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:49 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:53 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:55 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:57 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:08:59 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:09:01 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:09:03 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:09:05 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:09:07 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:10:05 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:18:44 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:18:51 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:18:56 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:18:58 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:01 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:03 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:05 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:07 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:09 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:11 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:13 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:15 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:17 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:19 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:21 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:23 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:25 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:27 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:29 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:31 icinga2-server icinga-vspheredb[25150]: [api aa.bb.cc.dd(id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:33 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:34 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'VM Quick Stats' is already running, skipping
    Jan 31 09:19:35 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:37 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:39 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:41 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:43 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:45 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:49 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:51 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:53 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:56 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:58 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:19:59 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:28:32 icinga-server icinga-vspheredb[25150]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Watchdog timeout (limit 10s)!
    Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25150 (icingacli) with signal SIGABRT.
    Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25151 (icingacli) with signal SIGABRT.
    Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Main process exited, code=killed, status=6/ABRT
    Jan 31 09:28:42 icinga-server systemd[1]: icinga-vspheredb.service: Failed with result 'watchdog'.
    Jan 31 09:29:13 icinga-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
    Jan 31 09:29:13 icinga-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1212.
    Jan 31 09:29:13 icinga-server systemd[1]: Stopped Icinga vSphereDB Daemon.
    Jan 31 09:29:13 icinga-server systemd[1]: Starting Icinga vSphereDB Daemon...
    Jan 31 09:29:14 icinga-server systemd[1]: Started Icinga vSphereDB Daemon.
    Jan 31 09:29:14 icinga-server icinga-vspheredb[25268]: [configwatch] DB configuration loaded
    Jan 31 09:29:14 icinga-server icinga-vspheredb[25268]: [db] sending DB config to child process
    Jan 31 09:29:15 icinga-server icinga-vspheredb[25268]: [db] Running DB cleanup (this could take some time)
    Jan 31 09:29:15 icinga-server icinga-vspheredb[25268]: [db] DB has been cleaned up
    Jan 31 09:29:15 icinga-server icinga-vspheredb[25268]: [localdb] ready
    Jan 31 09:29:15 icinga-server icinga-vspheredb[25268]: [api] launching server 2: vCenterId=1: https://user@aa.bb.cc.dd
    Jan 31 09:29:16 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Logged out
    Jan 31 09:29:16 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
    Jan 31 09:29:20 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:22 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:24 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:27 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:28 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:30 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:34 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:36 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:38 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:40 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:42 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:44 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:29:46 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:30:47 icinga-server icinga-vspheredb[25268]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Watchdog timeout (limit 10s)!
    Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25268 (icingacli) with signal SIGABRT.
    Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25269 (icingacli) with signal SIGABRT.
    Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Main process exited, code=killed, status=6/ABRT
    Jan 31 09:39:25 icinga-server systemd[1]: icinga-vspheredb.service: Failed with result 'watchdog'.
    Jan 31 09:39:55 icinga-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
    Jan 31 09:39:55 icinga-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1213.
    Jan 31 09:39:55 icinga-server systemd[1]: Stopped Icinga vSphereDB Daemon.
    Jan 31 09:39:56 icinga-server systemd[1]: Starting Icinga vSphereDB Daemon...
    Jan 31 09:39:59 icinga-server systemd[1]: Started Icinga vSphereDB Daemon.
    Jan 31 09:39:59 icinga-server icinga-vspheredb[25400]: [configwatch] DB configuration loaded
    Jan 31 09:39:59 icinga-server icinga-vspheredb[25400]: [db] sending DB config to child process
    Jan 31 09:40:00 icinga-server icinga-vspheredb[25400]: [db] Running DB cleanup (this could take some time)
    Jan 31 09:40:00 icinga-server icinga-vspheredb[25400]: [db] DB has been cleaned up
    Jan 31 09:40:00 icinga-server icinga-vspheredb[25400]: [localdb] ready
    Jan 31 09:40:00 icinga-server icinga-vspheredb[25400]: [api] launching server 2: vCenterId=1: https://user@aa.bb.cc.dd
    Jan 31 09:40:01 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Logged out
    Jan 31 09:40:02 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
    Jan 31 09:40:08 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:10 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:12 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:14 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:18 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:20 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:22 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:24 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:26 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:28 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:40:30 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:41:32 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:49:03 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:50:08 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:50:15 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:50:19 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:50:21 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:50:23 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:50:25 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:50:27 icinga-server icinga-vspheredb[25400]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:50:29 icinga-server icingacli[25400]: ERROR: ErrorException in /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Daemon/RemoteApi.php:103 with message: shell_exec(): Unable to execute 'id -G 33'
    Jan 31 09:50:30 icinga-server systemd[1]: icinga-vspheredb.service: Main process exited, code=exited, status=1/FAILURE
    Jan 31 09:50:30 icinga-server systemd[1]: icinga-vspheredb.service: Failed with result 'exit-code'.
    Jan 31 09:51:00 icinga-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
    Jan 31 09:51:00 icinga-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1214.
    Jan 31 09:51:00 icinga-server systemd[1]: Stopped Icinga vSphereDB Daemon.
    Jan 31 09:51:00 icinga-server systemd[1]: Starting Icinga vSphereDB Daemon...
    Jan 31 09:51:02 icinga-server systemd[1]: Started Icinga vSphereDB Daemon.
    Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [configwatch] DB configuration loaded
    Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [db] sending DB config to child process
    Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [db] Running DB cleanup (this could take some time)
    Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [db] DB has been cleaned up
    Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [localdb] ready
    Jan 31 09:51:02 icinga-server icinga-vspheredb[25495]: [api] launching server 2: vCenterId=1: https://user@aa.bb.cc.dd
    Jan 31 09:51:03 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Logged out
    Jan 31 09:51:03 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
    Jan 31 09:51:10 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:12 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:14 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:17 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:19 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:23 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:25 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:27 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:29 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:31 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:33 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:35 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:37 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:51:39 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:52:35 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 09:54:05 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:12 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:17 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:19 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:21 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:23 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:25 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:27 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:29 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:31 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:33 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:35 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:37 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:39 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:41 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:43 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:45 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:47 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:49 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:51 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:53 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:55 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:01:57 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:01 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:03 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:03 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'VM Quick Stats' is already running, skipping
    Jan 31 10:02:05 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:07 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:09 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:11 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:13 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:15 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:17 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:02:19 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:03:18 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:06:18 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:09:19 icinga-server icinga-vspheredb[25495]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Watchdog timeout (limit 10s)!
    Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25495 (icingacli) with signal SIGABRT.
    Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Killing process 25498 (icingacli) with signal SIGABRT.
    Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Main process exited, code=killed, status=6/ABRT
    Jan 31 10:11:13 icinga-server systemd[1]: icinga-vspheredb.service: Failed with result 'watchdog'.
    Jan 31 10:11:43 icinga-server systemd[1]: icinga-vspheredb.service: Service RestartSec=30s expired, scheduling restart.
    Jan 31 10:11:43 icinga-server systemd[1]: icinga-vspheredb.service: Scheduled restart job, restart counter is at 1215.
    Jan 31 10:11:43 icinga-server systemd[1]: Stopped Icinga vSphereDB Daemon.
    Jan 31 10:11:43 icinga-server systemd[1]: Starting Icinga vSphereDB Daemon...
    Jan 31 10:11:45 icinga-server systemd[1]: Started Icinga vSphereDB Daemon.
    Jan 31 10:11:45 icinga-server icinga-vspheredb[25645]: [configwatch] DB configuration loaded
    Jan 31 10:11:45 icinga-server icinga-vspheredb[25645]: [db] sending DB config to child process
    Jan 31 10:11:46 icinga-server icinga-vspheredb[25645]: [db] Running DB cleanup (this could take some time)
    Jan 31 10:11:46 icinga-server icinga-vspheredb[25645]: [db] DB has been cleaned up
    Jan 31 10:11:46 icinga-server icinga-vspheredb[25645]: [localdb] ready
    Jan 31 10:11:46 icinga-server icinga-vspheredb[25645]: [api] launching server 2: vCenterId=1: https://user@aa.bb.cc.dd
    Jan 31 10:11:47 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Logged out
    Jan 31 10:11:47 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
    Jan 31 10:11:52 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:11:54 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:11:56 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:11:58 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:12:02 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:12:04 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:12:06 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
    Jan 31 10:12:08 icinga-server icinga-vspheredb[25645]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping

Thank you!

Thomas-Gelf commented

Your daemon is dying again and again, being killed by systemd because of watchdog timeouts. Is your database running normally? As you mentioned a full disk, has the disk been full? Anything special about this setup? Very large vCenter, slow DB (containerized, object store, cloud service)?

Please restart the daemon, wait 10 seconds - then run systemctl status icinga-vspheredb.service, and share the output. Also, a SHOW PROCESSLIST; for your DB would be great, also executed about 10 seconds in. Please share some rough details (how many processes, are there any blocking ones or other special states) - and especially the processes related to vSphereDB.

david-ungurean commented

I think I found the problem.
I run free -m and notice that the server is not having enough memory (he only had 1 GB of RAM), so I added 3 GB of RAM, and now seems to be ok.

I suppose this will solve the problem.
Anyhow, I deleted the tmp files and restarted all the services.

The results after reboot:

image

Jan 31 13:22:09 icinga-server icinga-vspheredb[598]: [db] DB has been cleaned up
Jan 31 13:22:09 icinga-server icinga-vspheredb[598]: [localdb] ready
Jan 31 13:22:09 icinga-server icinga-vspheredb[598]: [api] launching server 2: vCenterId=1: https://usero@aa.bb.cc.dd
Jan 31 13:22:12 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Cookies changed, storing new ones
Jan 31 13:22:17 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:22:20 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:22:24 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:22:28 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:22:30 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping
Jan 31 13:23:42 icinga-server icinga-vspheredb[598]: [api aa.bb.cc.dd (id=2)] Task 'Events' is already running, skipping

Thank you!

David

Thomas-Gelf commented

You're welcome!