JMX exporter high memory usage.
harshal-choudhari-9393 opened this issue · 3 comments
I'm employing an independent JMX exporter in conjunction with Kafka Connect to retrieve Kafka Connect metrics in Prometheus within a Kubernetes pod. Specifically, I'm utilizing version 0.20 of the JMX exporter. With approximately 9000 topics in Kafka, I've allocated 5G of memory to the pod. Upon initiating the JMX exporter, JMX memory utilization promptly surges to 100%, leading to the termination of the pod and consequently halting Kafka Connect. I retrieved high cardinality metrics and attempted to mitigate this issue by deactivating them using the provided configuration However, I am still receiving all the metrics that I have attempted to disable.
excludeObjectNames: ["kafka.log:type=Log, name=Size,*"......]
Could you please guide me on reducing memory usage for the JMX exporter and disabling unnecessary metrics? Also, Let me know if any further information is needed.
@harshal-choudhari-9393 can you provide your JMX Exporter YAML configuration file?
Sure. Here is the YAML for the JMX exporter.
lowercaseOutputName: true
excludeObjectNames: ["kafka.server:type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec,*","kafka.cluster:type=Partition, name=UnderReplicated,*", "kafka.log:name=Size,type=Log,*","kafka.cluster:type=Partition,name=ReplicasCount,*"]
rules:
# Special cases and very specific rules
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
topic: "$4"
partition: "$5"
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
broker: "$4:$5"
- pattern: kafka.coordinator.(\w+)<type=(.+), name=(.+)><>Value
name: kafka_coordinator_$1_$2_$3
type: GAUGE
# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
#
# Note that these are missing the '_sum' metric!
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
quantile: "0.$8"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3_value
type: GAUGE
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>OneMinuteRate
name: kafka_$1_$2_$3_rate
type: GAUGE
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
quantile: "0.$6"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
name: kafka_$1_$2_$3_value
type: GAUGE
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>OneMinuteRate
name: kafka_$1_$2_$3_rate
type: GAUGE
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
quantile: "0.$4"```
@harshal-choudhari-9393 Can you provide your solution/resolution for future information?