Regression : Missing collectd_mcac_micros_bucket_ Metric in Cassandra 4.1
Nassz opened this issue · 1 comments
Description:
With the release of Cassandra 4.1, we have noticed a regression in the metric collection process for the collectd_mcac_micros_bucket_ metric, which is essential for monitoring and performance analysis. The MCAC module is no longer generating this metric as expected.
Steps to Reproduce:
Use Cassandra version 4.1+.
curl http://localhost:9103/metrics | grep micros_
Expected Behavior:
The collectd_mcac_micros_bucket_ metric should be generated and updated as before in Cassandra 4.1, allowing proper monitoring.
Environment:
- Cassandra Version: 4.1.0
- MCAC version : 0.3.4
Just for adding information, this issue is still present with more recent Cassandra 4.1 (tested 4.1.5) and the MCAC 0.3.4-beta1
I captured the following error in the Cassandra 4.1 logs which looks like it's falling in the ballpark of the issue:
INFO [insights-6-1] 2024-09-17 15:15:32,037 NoSpamLogger.java:105 - Not able to get buckets for org.apache.cassandra.metrics.dropped_message.internal_dropped_latency.finalize_propose_msg 128 type org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir$EstimatedHistogramReservoirSnapshot
There's an existing partial fix in k8ssandra for metrics for this in k8ssandra/management-api-for-apache-cassandra#428 as it seems there's a change in the bucket offsets in Cassandra 4.1, but it doesn't appear to be sufficient as per k8ssandra/management-api-for-apache-cassandra#444 and k8ssandra/management-api-for-apache-cassandra#500
The port for existing fix based on a quick research of the EstimatedHistogramReservoirSnapshot would likely go here in MCAC