/kafka-node

Experimenting with kafka

Primary LanguageJavaScript

kafka-node

This repo demonstrates a simple Node.js app that is using kafka with Kafka.js.

The producer has a topic named analytics. This topic has two partition. We want to store data coming in producer based on data content. If the data consists of all digit then we want to store it in partition 0 otherwise in partition 1.

image

The app has the following structure:

├── app
│   ├── admin.js
│   ├── client.js
│   ├── consumer.js
│   ├── node_modules
│   ├── package.json
│   ├── pnpm-lock.yaml
│   └── producer.js
├── docker-compose.yml

Please note that the following code in producer.js is used for creating command-line interfaces (CLIs) in Node.js, allowing users to input data interactively through the console.

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

Similarly, in consumer.js the following code is used to take command line arguement from the console.

const group = process.argv[2];

These are not specific to kafka implementation

Running locally

  • Initial setup
git clone git@github.com:NafiAsib/kafka-node.git
docker compose up -d # make sure to change IP to your device's private IP in docker-compose.yml
cd app
pnpm i
node admin.js

node-admin

  • Start Producer & start sending message
node producer.js

This'll start an interactive shell session. Type your messages and press Enter

  • You can start consumers with the following command
node consumer.js <consumer-group-name>

If you run multiple consumer with same <consumer-group-name>, kafka will start auto balancing.

Demo

image

Here we have two consumer group, group-1 & group-2. group-1 has only one consumer, thus all message are going to that consumer. group-2 has two consumer. So, one consumer is consuming partition 0 another one is consuming partition 1

image