dibbhatt/kafka-spark-consumer

Released latest version 1.0.8 of the Receiver

dibbhatt opened this issue 8 years ago · 0 comments

dibbhatt commented 8 years ago

Released latest version 1.0.8 of Receiver based Kafka Consumer for Spark Streaming.

Receiver is compatible with Kafka versions 0.8.x, 0.9.x and 0.10.x and All Spark Versions

Available at Spark Packages : https://spark-packages.org/package/dibbhatt/kafka-spark-consumer

Salient Features :

End to End No Data Loss without Write Ahead Log
ZK Based offset management for both consumed and processed offset
No dependency on WAL and Checkpoint for recovery on Driver failure
In-built PID Controller for Rate Limiting and Backpressure management
Custom Message Interceptor

Please refer to https://github.com/dibbhatt/kafka-spark-consumer/blob/master/README.md for more details