Low water mark reached. Dropping 100% of metrics.
prabinsh opened this issue · 5 comments
Hello,
I am seeing the following errors in cassandra-collectd.log
2021-03-01 10:45:24] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2021-03-01 10:45:34] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2021-03-01 10:45:44] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2021-03-01 10:45:49] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2021-03-01 10:45:50] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2021-03-01 10:45:51] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2021-03-01 10:45:54] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
I am also am running into this. I found https://support.datastax.com/hc/en-us/articles/360007958578-Missing-metrics-and-high-collectd-CPU-usage but it hasn't helped me yet.
I increase the write and read threads to 10 and didn’t help. Are you getting the same error in Prometheus? @harrison-tarr
is there a prometheus log? where do you see that error? i didn't look at prometheus much after i saw that grafana wasn't receiving any data and i found that log entry
Check targets in Prometheus
@prabinsh thanks for the note - it turns out since i'm running this on AWS i had to change the URL for the collectd service in prometheus. I don't have data for every graph now, but it does seem to be displaying some metrics in grafana now. looks like i'm still seeing the "low water mark reached" log message though