Simple kafka promise
This library is based on node-rdkafka and aim to provide a simple producer
and consumer wrapped with promise to allow the use of async / await
with minimal configuration and overhead.
This library is fully written in TypeScript.
Latest release 2.0.0 - 2020-01-14
Added
- (consumer) added consumer interface
Changed
- (index) changed export style to typescript
- (producer) migration of high level producer to typescript
- (consumer) migration of kafka consumer to typescript
- (mock) migration of mocked consumer to typescript
- (mock) migration of mocked producer to typescript
- (examples) updated examples for producer and consumer
Fixed
- (producer) fixed thrown error if kafka.producer.topicsPrefix was missing
Removed
- (producer) removed kafkaNProducer
- (producer) removed poll() before disconnecting
- (dependencies) removed unused hirestime and uuid dependency
- (dependencies) removed unused prom-client, s3pweb-logger and config dependency
Breaking changes from 1.x.x
- Producer and consumer now are classes and have a constructor
- No more direct need for node-config and a logger
Installation
npm install @s3pweb/simple-kafka-promise
Configuration
Minimal configuration for the consumer is:
{ "metadata.broker.list": "0.0.0.0:9094", "group.id": "test.group" }
Minimal configuration for the producer is:
{ "metadata.broker.list": "0.0.0.0:9094" }
This project is based on node-rdkafka and supports the same configuration options. Go here for more details.
Create a consumer instance
const KafkaConsumer = require('@s3pweb/simple-kafka-promise').KafkaConsumer
// Create a new instance
const consumer = new KafkaConsumer({ 'metadata.broker.list': '0.0.0.0:9094', 'group.id': 'test.group' }, 1000)
// Connect
await consumer.connect(['topicName'])
// Consume messages
const messagesArray = await consumer.listen(100, true)
// Disconnect the consumer
await consumer.disconnect()
To use with typescript, just change the import to
import { KafkaConsumer } from '@s3pweb/simple-kafka-promise';
Create a producer instance
const KafkaProducer = require('@s3pweb/simple-kafka-promise').KafkaProducer
// Create a new instance
const producer = new KafkaProducer({ 'metadata.broker.list': '0.0.0.0:9094' }, '')
// Connect
await producer.connect(['topicName'])
// Produce some messages
const offset = await producer.sendMessage(topicName, { message: `My message.` }, 0, null)
// Disconnect
await producer.disconnect()
To use with typescript, just change the import to
import { KafkaProducer } from '@s3pweb/simple-kafka-promise';
Example
To produce some messages take a look at ./examples/producer.js
and to consume some messages take a look at ./examples/consumer.js
.
If you have docker, you can use ./examples/docker-compose.yaml
to start one zookeeper
and one kafka
stack on your machine.
This stack comes with kafkamanager
and kafkadrop
for easy monitoring and debugging.
Docker
If you want to build a docker image based on alpine linux, you need to add some packages to the base image. Go here for more details.