/kafkajs-metrics

A lightweight KafkaJS library that provides simple access to key metrics and real-time alerts within your code

Primary LanguageJavaScriptMozilla Public License 2.0MPL-2.0

KafkaJSMetrics

A lightweight KafkaJS library that provides simple access to key metrics within your code. KafkaJSMetrics enables custom breakpoint alerts and logging of key metrics, allowing for client health to be monitored within applications. The optional visualizer feature introduces a real-time, web browser display of the metrics associated with your KafkaJS client.

Table of Contents

Introduction

Apache Kafka is a distributed event streaming platform that is designed to handle high performance data pipelines in real-time. KafkaJS is a lightweight, user-friendly Apache Kafka client library that runs on Node.js, providing a simple and reliable way to produce and consume messages from Kafka clusters. Event emitters called instrumentation events are built into KafkaJS and provide information about the performance and behavior of a user's Kafka producers and consumers. These events are designed to help developers monitor and troubleshoot their Kafka applications. However, incorporating instrumentation events into code can be tedious and repetitive.

KafkaJSMetrics is a continuously-growing open source product that is designed to simplify and shorten the metric exposing processes. By committing to 100% Javascript alongside KafkaJS, the library leverages already built-in instrumentation events without relying on other third-party applications or requiring any additional installations. The library gives developers simple access to key metrics required to monitor the health of a Kafka instance.

Features

Below are some of the key features from KafkaJSMetrics. For a full explanation of library functionality, visit our docs.

Key features:

  • Access to key consumer/producer/admin metrics, including
    • offset lag
    • messages consumed
    • total number of consumer/producer/admin currently connected
    • heartbeat
    • partition assignment
    • see docs for complete metrics
  • Custom Breakpoint Alerts and On/Off metrics logging options, including
    • heartbeat monitor
    • request queue size monitor
    • request pending duration monitor
    • offset lag monitor
    • see docs for complete Breakpoint Alert and metrics logging options
  • Web browser visualizer displays consumer metrics in real-time
    • Secure token access and developer-friendly interface

Getting Started

KafkaJSMetrics works with your KafkaJS client. For information on setting up your KafkaJS client, visit KafkaJS.

  • Installation and Usage walk through setting up the KafkaJSMetrics library for use within your KafkaJS Client code

  • Visualization takes you through the optional feature set-up for web browser visual display of your KafkaJS Client metrics

Installation

Install the kafkajs-metrics-visualize package from npm

npm install kafkajs-metrics-visualize

Usage

KafkaJSMetrics works with your existing KafkaJS client. More information about KafkaJS at kafka.js.org.

  1. Require KafkaJSMetrics in your KafkaJS Client file to access metricize function

  1. Pass your KafkaJS Client instance into the metricize function (sample KafkaJS Client provided)

    • All consumer, producer, and admin instances created from this client will now have metrics and metrics-associated functionality

  1. Run your KafkaJS Client file (skip this step if you are planning use the visualizer)
node client.js

Turn on logging methods

Visit the docs for a full list of available logging methods

Example:

Prints to console:

Turn on breakpoint alerts

Visit the docs for a full list of available breakpoint alert methods

Example:

Prints to console:

Visualization

The following steps will allow you to access the web browser visualizer to display metrics. This visualizer is an optional feature and will not affect the library performance.

  1. Open the KafkaJSMetrics webpage. Click on the visualizer tab and generate a new token

  1. Copy the token and head back to your KafkaJS client file. Inside the metricize function, add two additional arguments:

    • first argument is the boolean value true
    • second argument is your copied token

  1. Run your KafkaJS Client file
node client.js
  1. Back in the browser, click authenticate to access your web browser metrics display

Community

As KafkaJSMetrics continues to grow, we welcome and encourage feedback! We are also always trying to improve our code so we actively welcome all contributions! If you would like to be a part of this product, please follow the steps below.

  1. Fork this repo of KafkaJSMetrics and then clone it to your machine

  2. Check out into the branch that you want to make contributions in

    1. Contributions to the website and/or visualizer:

      git checkout visualizer

    2. Contributions to the library:

      git checkout library

  3. Create a feature branch

git checkout -b [feature/AmazingFeature]
  1. Add all your changes with the add command
git add [all your changes]
  1. Commit all your changes
git commit -m '<your comment>'

  1. Make sure the visualizer OR library branch is up to date

    1. Check back out to the visualizer OR library branch
    git checkout visualizer [OR] library
    1. Pull down the latest version
    git pull origin visualizer [OR] library
    1. Check back out to your feature branch
    git checkout [feature/AmazingFeature]

  2. Merge your branch with visualizer OR library

git merge visualizer [OR] library
  1. Resolve any merge conflicts
  2. Run unit tests and make sure all tests pass
npm test
  1. Push up your branch with the command
git push origin [feature/AmazingFeature]
  1. Finally, open a pull request for us to review!

Contributors

Other

Important links:

KafkaJSMetrics and KafkaJSMetrics Visualizer are licensed under the Mozilla Public License version 2.0 and are available for free. We have no affiliation with and are not endorsed by either The Apache Software Foundation or the developers of KafkaJS.