Kafka NightOwl is an open-source project dedicated to developing a powerful Kafka cluster visualization tool specifically designed for senior software developers. Our mission is to empower developers with a simple yet powerful tool that allows them to monitor and visualize their Kafka clusters effortlessly and get live time alerts which enable developers to effectively troubleshoot, optimize performance, and efficiently manage their Kafka clusters.
Real-time Metrics and Charts: Kafka NightOwl provides you with real-time insights into your Kafka clusters' health and performance through easy-to-understand metrics and visually appealing charts. Gain a deeper understanding of topics, partitions, brokers, and more.
Seamless Prometheus Integration: Setting up Prometheus to scrape Kafka metrics has never been easier. Kafka NightOwl guides you through the process, ensuring that your data collection is a smooth experience.
Intuitive Dashboard: The application's main dashboard serves as your central hub for Kafka cluster monitoring. Our integration of the ChartJS library provides numerous customization features to the dashboard including moving, resizing, and auto-scale and fit.
Alerts and Notifications: Kafka NightOwl leverages Prometheus AlertManager to keep you informed about critical issues. Receive alerts about errors, performance drops, and other anomalies, allowing you to take proactive measures.
-
Creating a User Account Before diving into Kafka cluster visualization, you'll need to create a user account. Simply sign up with your email and password to access the powerful features of Kafka NightOwl.
-
Connecting to Prometheus To kickstart the monitoring process, connect Kafka NightOwl to your Prometheus instance running locally. Our team has provided sample configuration files for Kafka, Prometheus, and AlertManager as well as a simple guide to connect these together.
-
Exploring Metrics Once connected, Kafka NightOwl provides an intuitive interface to explore various Kafka metrics. Select topics, partitions, or brokers, and watch as the application generates insightful charts and graphs, giving you a real-time view of your cluster's health. Customize your metrics display by dragging and resizing charts and even add more metrics to your dashboard.
-
Staying Informed with Alerts Kafka NightOwl's AlertManager integration empowers you to stay ahead of critical issues. Whenever an anomaly or error is detected, an alert will be triggered. You can acknowledge and manage these alerts directly from the application.
- Setting up Kafka by following Apache Kafka Quickstart tutorial: Apache Kafka Quickstart
- Configure JMX Exporter for Kafka
- Download the JMX Exporter jar file Here
- Configure the JMX exporter configuration file for Kafka (example configuration files can be found Here)
- CD to your Kafka directory
- Run
java -jar jmx_prometheus_httpserver-0.19.0.jar <port number> <exporter-config-file-path>
to expose your metrics at localhost:[port number]/metrics
- Run
export KAFKA_OPTS="-javaagent:/<exporter-jar-file-path>/jmx_prometheus_javaagent-0.19.0.jar=<port number>:/<kafka-yml-file-path>/kafka-2_0_0.yml"
- Start your Kafka server
- Setting up Prometheus by following: Prometheus Tutorial
- Configure the prometheus.yml file as shown in /setup/prometheus
- Download and Setup AlertManager from Prometheus
- Configure the alertManager.yml file as shown in /setup/prometheus
| Developed By | Github | |
|---|---|---|
| Kelvin Chen |  |
 |
| Jeremy Holland | |
|
| Paul Kim | |
|
| Carlos Revilla | |
|
| Colin Silvers | |
|
This project is licensed under the MIT License