This repo demonstrates examples of JMX monitoring stacks that can monitor Confluent Cloud and Confluent Platform. While Confluent Cloud UI and Confluent Control Center provides an opinionated view of Apache Kafka monitoring, JMX monitoring stacks serve a larger purpose to our users, allowing them to setup monitoring across multiple parts of their organization, many outside of Kafka, and to have a single pane of glass.
This project provides metrics and dashboards for:
- Confluent Platform with Prometheus and Grafana
- Confluent Platform on Kubernetes with Prometheus and Grafana
- Confluent Platform with New Relic
- Confluent Platform with Prometheus, Metricbeat and Kibana
- Confluent Platform with Jolokia Agent
- Confluent Platform with Jolokia, Telegraf and InfluxDB
- Confluent Platform with Datadog agent
- Confluent Cloud with Prometheus and Grafana
- Confluent Cloud with Metricbeat and Kibana
- Confluent Cloud with Opentelemetry and New Relic
Some examples:
List of available dashboards for Confluent Platform:
Dashboard | Prometheus and Grafana | New Relic | Metricbeat and Kibana | Telegraf and Influx | Datadog |
---|---|---|---|---|---|
Kafka Cluster | yes | yes | yes | yes | yes |
Zookeeper | yes | yes | yes | ||
KRaft | yes | ||||
Schema Registry | yes | yes | |||
Kafka Connect | yes | yes | |||
ksqlDB | yes | yes | |||
Producer/Consumer | yes | yes | yes | yes | |
Lag Exporter | yes | ||||
Topics | yes | yes | |||
Kafka Streams | yes | ||||
Kafka Streams RocksDB | yes | ||||
Quotas | yes | ||||
TX Coordinator | yes | ||||
Rest Proxy | yes | ||||
Cluster Linking | yes | ||||
Oracle CDC connector | yes | ||||
Debezium connectors | yes | ||||
Mongo connector | yes | ||||
librdkafka clients | yes | ||||
Confluent RBAC | yes | ||||
Replicator | yes | ||||
Tiered Storage | yes |
List of available dashboards for Confluent Cloud:
Dashboard | Prometheus and Grafana | New Relic | Metricbeat and Kibana |
---|---|---|---|
Cluster | yes | yes | yes |
Producer/Consumer | yes | ||
ksql | yes | ||
Billing/Cost tracking | yes |
Alerts are available for the stacks:
- jmxexporter-prometheus-grafana including alerts on broker, zookeeper and kafka connect.
To add JMX exporter configurations to Confluent cp-ansible, please refer to this README
To add JMX exporter configurations to your Kubernetes workspace, please refer to this README
This repo is intended to work smoothly with Confluent cp-demo.
Make sure you have enough system resources on the local host to run this. Verify in the advanced Docker preferences settings that the memory available to Docker is at least 8 GB (default is 2 GB).
NOTE: jq is required to be installed on your machine to run the demo.
-
Ensure that cp-demo is not already running on the local host.
-
Decide which monitoring stack to demo: and set the
MONITORING_STACK
variable accordingly.
NOTE: New Relic requires a License Key to be added in jmxexporter-newrelic/start.sh
NOTE: Datadog requires a DATADOG_API_KEY and DATADOG_SITE to be added in datadog/start.sh. Datadog offers 14 day trial for new users.
# Set only one of these
MONITORING_STACK=jmxexporter-prometheus-grafana
MONITORING_STACK=metricbeat-elastic-kibana
MONITORING_STACK=jmxexporter-newrelic
MONITORING_STACK=jolokia
MONITORING_STACK=jolokia-telegraf-influxdb
MONITORING_STACK=datadog
- Clone
cp-demo
and checkout a branch.
# Example with CP-DEMO 7.6.1 version (all branches starting from 7.2.0 have been tested)
CP_DEMO_VERSION=7.6.1-post
[[ -d "cp-demo" ]] || git clone https://github.com/confluentinc/cp-demo.git
(cd cp-demo && git fetch && git checkout $CP_DEMO_VERSION && git pull)
- Clone
jmx-monitoring-stacks
and checkout main branch.
[[ -d "jmx-monitoring-stacks" ]] || git clone https://github.com/confluentinc/jmx-monitoring-stacks.git
(cd jmx-monitoring-stacks && git fetch && git checkout main && git pull)
- Start the monitoring solution with the STACK selected. This command also starts cp-demo, you do not need to start cp-demo separately.
${MONITORING_STACK}/start.sh
- Stop the monitoring solution. This command also stops cp-demo, you do not need to stop cp-demo separately.
${MONITORING_STACK}/stop.sh
For an example that showcases how to monitor Apache Kafka client applications, and steps through various failure scenarios to see how they are reflected in the provided metrics, see the Observability for Apache Kafka® Clients to Confluent Cloud tutorial.
To run a lightweight dev environment:
cd dev-toolkit
- Put your new dashboards into the
grafana-wip
folder start.sh
-> It will create a minimal environment with a KRaft cluster, prometheus, grafana and a spring based java client- For Grafana, go to http://localhost:3000, login with admin/password
stop.sh
To add more use cases, we are leveraging the docker profiles.
To run replicator scenario, i.e. start.sh --profile replicator
. It's possible to combine profiles as well, i.e. start.sh --profile schema-registry --profile ksqldb
.
Currently supported profiles:
- replicator
- schema-registry
- ksqldb
- consumer (with kafka-lag-exported included)
- clientsreduced (kafka clients with a limited number of metrics exposed)
- What if I need more components?
More docker-compose envs will be released in the future, in the meantime you can use Kafka Docker Composer
- What if I need more prometheus jobs?
You can add them to the start.sh
, i.e.
# ADD client monitoring to prometheus config
cat <<EOF >> assets/prometheus/prometheus-config/prometheus.yml
- job_name: 'spring-client'
static_configs:
- targets: ['spring-client:9191']
labels:
env: "dev"
EOF
You can also change the prometheus configuration here.