Performance data writer causes runaway-series-cardinality in InfluxDB
slalomsk8er opened this issue · 0 comments
slalomsk8er commented
Expected Behavior
Performance data writer is "setup and forget" and provides a clear benefit.
Current Behavior
- In large and highly dynamic vCenters (virtual desktop infrastructure) the writer can result in runaway-series-cardinality:
Sep 15 12:02:11 icingalp06 influxd-systemd-start.sh[6778]: ts=2023-09-15T10:02:11.985985Z lvl=warn msg="max-values-per-tag limit may be exceeded soon" log_id=0kFnS2vG000 service=store perc=100% n=100000 max=100000 db_instance=icinga_vspheredb_ictvdivclp01 measurement=VmMemory tag=vm_uuid
[12:03:35 root@icingalp06 ~]$ journalctl -xeu influxdb | grep max-values-per-tag | cut -d' ' -f 19 | sort -u
db_instance=icinga_vspheredb_ictvdivclp01
[12:04:08 root@icingalp06 ~]$ journalctl -xeu influxdb | grep max-values-per-tag | cut -d' ' -f 20 | sort -u
measurement=VirtualDisk
measurement=VirtualNetworkAdapter
measurement=VmCpu
measurement=VmMemory
[12:04:15 root@icingalp06 ~]$ journalctl -xeu influxdb | grep max-values-per-tag | cut -d' ' -f 21 | sort -u
tag=instance
tag=vm_moref
tag=vm_uuid
https://www.influxdata.com/blog/solving-runaway-series-cardinality-when-using-influxdb/
- Also in the module itself, there isn't a clear benefit for setting all this up. Including "fixing" the Grafana dashboards.
So for the time being, we disabled the feature and droppend the InfluxDB icinga_vspheredb_*
instances.
Because this feature endangers the Grafana graphs in the monitoring module and I suspect nobody except me ever opened the Grafana dashboard belonging to this module.
Possible Solution
- Performance data writer toggle per vCenter
- Performance data writer sends data in a way to not trigger runaway-series-cardinality
- Add Grafana graphs to the module and not just display the current state of CPU, RAM and the like.
Your Environment
- VMware vCenter®/ESXi™-Version: vCenter Server 6.7.0
- Version/GIT-Hash of this module: 1.6.0
- Icinga Web 2 version: 2.11.4
- Operating System and version: AlmaLinux8
- Webserver, PHP versions: PHP 8.0.30