False positive "Cannot retrieve the datacenter name" with Cassandra v3.11.0 ?
rafi opened this issue · 0 comments
rafi commented
I'm seeing this NullPointerException
message with Cassandra v3.11.0 and exporter v2.3.5 running together in a pod with Kubernetes. This doesn't seem to affect the exporter's behavior, everything works as expected beside this error on startup:
(Could it be that exporter starts before cassandra and connection isn't available yet on startup?)
[main] INFO com.criteo.nosql.cassandra.exporter.Config - Loading yaml config from /etc/cassandra_exporter/config.yml
[main] ERROR com.criteo.nosql.cassandra.exporter.JmxScraper - Cannot retrieve the datacenter name information for the node
javax.management.RuntimeMBeanException: java.lang.NullPointerException
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
Relevant cassandra config:
root@cassandra-0# cassandra -v
3.11.0
root@cassandra-0# grep 'cluster_name\|seed\|class' /etc/cassandra/cassandra.yaml
cluster_name: Foobar
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
- seeds: "cassandra-0.cassandra-headless.default.svc.cluster.local"
- class_name: org.apache.cassandra.security.JKSKeyProvider
- class_name: org.apache.cassandra.net.RateBasedBackPressure
Exporter config:
root@cassandra-0:/# cat /etc/cassandra_exporter/config.yml
host: localhost:7199
ssl: False
user:
password:
listenAddress: 0.0.0.0
listenPort: 5556
# Regular expression to match environment variable names that will be added
# as labels to all data points. The name of the label will be either
# $1 from the regex below, or the entire environment variable name if no match groups are defined
#
# Example:
# additionalLabelsFromEnvvars: "^ADDL\_(.*)$"
additionalLabelsFromEnvvars:
blacklist:
# To profile the duration of jmx call you can start the program with the following options
# > java -Dorg.slf4j.simpleLogger.defaultLogLevel=trace -jar cassandra_exporter.jar config.yml --oneshot
#
# To get intuition of what is done by cassandra when something is called you can look in cassandra
# https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/metrics
# Please avoid to scrape frequently those calls that are iterating over all sstables
# Unaccessible metrics (not enough privilege)
- java:lang:memorypool:.*usagethreshold.*
# Leaf attributes not interesting for us but that are presents in many path
- .*:999thpercentile
- .*:95thpercentile
- .*:fifteenminuterate
- .*:fiveminuterate
- .*:durationunit
- .*:rateunit
- .*:stddev
- .*:meanrate
- .*:mean
- .*:min
# Path present in many metrics but uninterresting
- .*:viewlockacquiretime:.*
- .*:viewreadtime:.*
- .*:cas[a-z]+latency:.*
- .*:colupdatetimedeltahistogram:.*
# Mostly for RPC, do not scrap them
- org:apache:cassandra:db:.*
# columnfamily is an alias for Table metrics
- org:apache:cassandra:metrics:columnfamily:.*
# Should we export metrics for system keyspaces/tables ?
- org:apache:cassandra:metrics:[^:]+:system[^:]*:.*
# Logback doesn't have any useful metrics
- ch:qos:logback:.*
# Don't scrap us
- com:criteo:nosql:cassandra:exporter:.*
maxScrapFrequencyInSec:
50:
- .*
# Refresh those metrics only every hour as it is costly for cassandra to retrieve them
3600:
- .*:snapshotssize:.*
- .*:estimated.*
- .*:totaldiskspaceused:.*