[Bug]: Single quote in labels breaks charts from prometheus collector
sepek opened this issue · 1 comments
sepek commented
Bug description
Having a label in the style of e.g. operator="Isle d'Abeau" in a Prometheus label, e.g.:
my_special_exporter{id="r4nd0m", information="FOOBAR", operator="Isle d'Abeau", status="0"} 1714400258
will create a faulty chart which will simply cut of after the ':
prometheus_time_report.my_special_exporter-information=FOOBAR-id=r4nd0m-operator=Isle_d
So also the status is missing, additionally it will not contain the reported data, so the value is 0 instead of 1714400258
I couldn't find a definition which prohibits a ' in the label also it works fine in Prometheus so I suspect it's a bug in the Prometheus module of NetData
Expected behavior
Having the full chart, maybe with ' replaced by something else and including all reported labels and with actual values reported.
Steps to reproduce
- Create/Add a label with an
' - Check your charts for the label
Installation method
manual setup of official DEB/RPM packages
System info
Linux vps-44774cee 6.1.0-13-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
/etc/cloud-release:ID="genericcloud"
/etc/cloud-release:VERSION="20231004-1523"
/etc/os-release:PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
/etc/os-release:NAME="Debian GNU/Linux"
/etc/os-release:VERSION_ID="12"
/etc/os-release:VERSION="12 (bookworm)"
/etc/os-release:VERSION_CODENAME=bookworm
/etc/os-release:ID=debianNetdata build info
Packaging:
Netdata Version ____________________________________________ : v1.45.3
Installation Type __________________________________________ : binpkg-deb
Package Architecture _______________________________________ : x86_64
Package Distro _____________________________________________ :
Configure Options __________________________________________ : dummy-configure-command
Default Directories:
User Configurations ________________________________________ : /etc/netdata
Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
Permanent Databases ________________________________________ : /var/lib/netdata
Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
Static Web Files ___________________________________________ : /var/lib/netdata/www
Log Files __________________________________________________ : /var/log/netdata
Lock Files _________________________________________________ : /var/lib/netdata/lock
Home _______________________________________________________ : /var/lib/netdata
Operating System:
Kernel _____________________________________________________ : Linux
Kernel Version _____________________________________________ : 6.1.0-13-cloud-amd64
Operating System ___________________________________________ : Debian GNU/Linux
Operating System ID ________________________________________ : debian
Operating System ID Like ___________________________________ : unknown
Operating System Version ___________________________________ : 12 (bookworm)
Operating System Version ID ________________________________ : none
Detection __________________________________________________ : /etc/os-release
Hardware:
CPU Cores __________________________________________________ : 2
CPU Frequency ______________________________________________ : 2399000000
RAM Bytes __________________________________________________ : 4022026240
Disk Capacity ______________________________________________ : 85899345920
CPU Architecture ___________________________________________ : x86_64
Virtualization Technology __________________________________ : kvm
Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
Container __________________________________________________ : none
Container Detection ________________________________________ : systemd-detect-virt
Container Orchestrator _____________________________________ : none
Container Operating System _________________________________ : none
Container Operating System ID ______________________________ : none
Container Operating System ID Like _________________________ : none
Container Operating System Version _________________________ : none
Container Operating System Version ID ______________________ : none
Container Operating System Detection _______________________ : none
Features:
Built For __________________________________________________ : Linux
Netdata Cloud ______________________________________________ : YES
Health (trigger alerts and send notifications) _____________ : YES
Streaming (stream metrics to parent Netdata servers) _______ : YES
Back-filling (of higher database tiers) ____________________ : YES
Replication (fill the gaps of parent Netdata servers) ______ : YES
Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip)
Contexts (index all active and archived metrics) ___________ : YES
Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
Machine Learning ___________________________________________ : YES
Database Engines:
dbengine ___________________________________________________ : YES
alloc ______________________________________________________ : YES
ram ________________________________________________________ : YES
none _______________________________________________________ : YES
Connectivity Capabilities:
ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
static (Netdata internal web server) _______________________ : YES
h2o (web server) ___________________________________________ : YES
WebRTC (experimental) ______________________________________ : NO
Native HTTPS (TLS Support) _________________________________ : YES
TLS Host Verification ______________________________________ : YES
Libraries:
LZ4 (extremely fast lossless compression algorithm) ________ : YES
ZSTD (fast, lossless compression algorithm) ________________ : YES
zlib (lossless data-compression library) ___________________ : YES
Brotli (generic-purpose lossless compression algorithm) ____ : NO
protobuf (platform-neutral data serialization protocol) ____ : YES (system)
OpenSSL (cryptography) _____________________________________ : YES
libdatachannel (stand-alone WebRTC data channels) __________ : NO
JSON-C (lightweight JSON manipulation) _____________________ : YES
libcap (Linux capabilities system operations) ______________ : NO
libcrypto (cryptographic functions) ________________________ : YES
libyaml (library for parsing and emitting YAML) ____________ : YES
Plugins:
apps (monitor processes) ___________________________________ : YES
cgroups (monitor containers and VMs) _______________________ : YES
cgroup-network (associate interfaces to CGROUPS) ___________ : YES
proc (monitor Linux systems) _______________________________ : YES
tc (monitor Linux network QoS) _____________________________ : YES
diskspace (monitor Linux mount points) _____________________ : YES
freebsd (monitor FreeBSD systems) __________________________ : NO
macos (monitor MacOS systems) ______________________________ : NO
statsd (collect custom application metrics) ________________ : YES
timex (check system clock synchronization) _________________ : YES
idlejitter (check system latency and jitter) _______________ : YES
bash (support shell data collection jobs - charts.d) _______ : YES
debugfs (kernel debugging metrics) _________________________ : YES
cups (monitor printers and print jobs) _____________________ : YES
ebpf (monitor system calls) ________________________________ : YES
freeipmi (monitor enterprise server H/W) ___________________ : YES
nfacct (gather netfilter accounting) _______________________ : YES
perf (collect kernel performance events) ___________________ : YES
slabinfo (monitor kernel object caching) ___________________ : YES
Xen ________________________________________________________ : YES
Xen VBD Error Tracking _____________________________________ : NO
Logs Management ____________________________________________ : YES
Exporters:
AWS Kinesis ________________________________________________ : NO
GCP PubSub _________________________________________________ : NO
MongoDB ____________________________________________________ : YES
Prometheus (OpenMetrics) Exporter __________________________ : YES
Prometheus Remote Write ____________________________________ : YES
Graphite ___________________________________________________ : YES
Graphite HTTP / HTTPS ______________________________________ : YES
JSON _______________________________________________________ : YES
JSON HTTP / HTTPS __________________________________________ : YES
OpenTSDB ___________________________________________________ : YES
OpenTSDB HTTP / HTTPS ______________________________________ : YES
All Metrics API ____________________________________________ : YES
Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
Trace All Netdata Allocations (with charts) ________________ : NO
Developer Mode (more runtime checks, slower) _______________ : NOAdditional info
No response
sepek commented
Additional debug output of the Prometheus plugin:
my_special_exporter{id="r4nd0m", information="FOOBAR", operator="Isle d'Abeau", status="0"} 1714400258
prometheus_time_report.my_special_exporter-information=FOOBAR-id=r4nd0m-operator=Isle_d
[…]
CHART 'prometheus_time_report.my_special_exporter-information=FOOBAR-id=r4nd0m-operator=Isle_d'Abeau-status=0' '' 'DESCRIPTION' 'seconds' 'my_special_exporter' 'prometheus.time_report.my_special_exporter' 'line' '70000' '300' '' 'go.d' 'prometheus'
CLABEL 'information' 'FOOBAR' '1'
CLABEL 'id' 'r4nd0m' '1'
CLABEL 'operator' 'Isle d'Abeau' '1'
CLABEL 'status' '0' '1'
CLABEL '_collect_job' 'time_report' '1'
CLABEL_COMMIT
DIMENSION 'my_special_exporter' 'my_special_exporter' 'absolute' '1' '1000' ''
BEGIN 'prometheus_time_report.my_special_exporter-information=FOOBAR-id=r4nd0m-operator=Isle_d'Abeau-status=0'
SET 'my_special_exporter' = 1714400258000
END
[…]
The problem is clearly visible here, too many ':
CLABEL 'operator' 'Isle d'Abeau' '1'
as well as here:
BEGIN 'prometheus_time_report.my_special_exporter-information=FOOBAR-id=r4nd0m-operator=Isle_d'Abeau-status=0'