Prometheus setup - connect cassandracluster to an existing prometheus server

[mizunos]

Type of question

Are you asking about community best practices, how to implement a specific feature, or about general context and help around casskop ?
Help with setting up prometheus to monitor cassandra

Question

I already have a prometheus pod spinned up in the current Cassandra cluster (managed by casskop). Is the JMX exporter already deployed with Cassandracluster and if so what do I need to change on the prometheus.yaml to point to that exporter?,

The sample file is for service monitor and prometheus deployed using prometheus operator which is not our case

Any pointers is greatly appreciated. Other than that, casskop rocks

Environment

• casskop version:

  Release 0.5.6

• Kubernetes version information:

GKE 1.15.20

• Kubernetes cluster kind:

• Cassandra version: 3.11

Additional context
Add any other context about the question here.

[cscetbon]

@mizunos see https://orange-opensource.github.io/casskop/docs/2_setup/1_getting_started#monitoring. In short you'll need to create a new serviceMonitor with the correct settings and import our dashboard. You'll probably also need the install the plugins we use though or use your own Grafana dashboard. See https://github.com/Orange-OpenSource/casskop/tree/master/monitoring, I think the documentation might be slightly out of date. @fdehay We should probably check the documentation is up to date

[mizunos]

So the ServiceMonitor object that was defined in CassKop example is to be used if we were deploy Prometheus using the coreOS operator.
In this case, I am running Cortexmetrics which already have prometheus as part of the deployment. Looking through the code, it seemed like there is a service already deployed when I spinned up Cassandra using Casskop (cassandra-exporter-jmx on port 9500) and that the Cassandra exporter is part of the deployment of Cassandra container (according to your bootstrap docker file). So it seemed I only need to add the relabeling configuration in the service monitor yaml to the prometheus.yml of my current Prom server configmap to get the exporter and the current server to talk to each other(provided that I point the PROM server to the right scraping endpoint).

Am I thinking about this correctly?

[cscetbon]

Yes the service cassandra-exporter-jmx is already deployed, hence why I didn't talk about it 😬
If the service monitor works with your prometheus deployment, then you should be on the right path.

[mizunos]

The plan worked. I added the service monitor configuration job to the current prometheus server configmap, and reset the server. The new configmap now picked up the target cassandra node without any problem. Off to setup the grafana dashbaord now. Thanks

Quick note - if you have any information about setting up cassandra -reaper in K8s, would appreciate the information.