This project comprises a few helper modules for operating the Kafka Java Client Driver in a Scala codebase.
These modules are production ready, actively maintained and are used in a large scale production system.
To resolve any of the modules, add the following resolver to the build.sbt:
resolvers += Resolver.bintrayRepo("cakesolutions", "maven")A thin Scala wrapper over the official Apache Kafka Java Driver. This module is useful for integrating with Kafka for message consumption/delivery, but provides some helpers for convenient configuration of the driver and usage from Scala. Minimal 3rd party dependencies are added in addition to the Kafka client.
For configuration and usage, see the Wiki: Scala Kafka Client Guide
SBT library dependency:
libraryDependencies += "net.cakesolutions" %% "scala-kafka-client" % "2.3.1"This module provides a configurable asynchronous and non-blocking Kafka Consumer and Producer Actor implementations to support high performance, parallel custom stream processing in an Akka application. These components are specifically intended for use cases where high performance and scalable message processing is required with specific concern for message delivery guarantees and resilience.
For configuration and usage, see the Wiki: Akka Integration
SBT library dependency:
libraryDependencies += "net.cakesolutions" %% "scala-kafka-client-akka" % "2.3.1"The TestKit module provides some tools to support integration testing of client service code that depends on a running Kafka Server. Helps the setup of an in-process Kafka and Zookeeper server.
For usage, see the Wiki: TestKit User Guide
SBT library dependency:
libraryDependencies += "net.cakesolutions" %% "scala-kafka-client-testkit" % "2.3.1" % "test"Starting after version 0.8.0, the versioning for Scala Kafka client will be tracking Kafka's versioning scheme.
Binary compatibility in the new versioning system works as follows:
- The first and the second digit in the version indicate compatibility with the Kafka driver.
For example,
0.9.0.0is compatible with Kafka 0.9 and0.10.0.0is compatible with Kafka 0.10. - The third digit in the version indicates an incompatible change between Scala Kafka client versions.
For example,
0.9.0.1is not binary compatible with0.9.1.0. - The fourth digit in the version indicates a compatible change between Scala Kafka client versions.
For example,
0.9.0.0is compatible with0.9.0.1.
Both the 0.10.* and 1.0.* versions are maintained concurrently.
Here is the full table of binary compatibilities between Scala Kafka client and the Kafka Java driver:
| Scala Kafka client | Kafka Java Driver |
|---|---|
| 2.3.1 | 2.3.1 |
| 2.1.0 | 2.1.0 |
| 2.0.0 | 2.0.0 |
| 1.1.1 | 1.1.1 |
| 1.0.0 | 1.0.0 |
| 0.11.0.0 | 0.11.0.0 |
| 0.10.2.x | 0.10.2.x |
| 0.10.1.x | 0.10.1.x |
| 0.10.0.0 | 0.10.0.x |
| 0.9.0.0 | 0.9.0.x |
| 0.8.0 | 0.10.0.0 |
| 0.7.0 | 0.9.0.1 |
- Update to Kafka 2.3.1
- Update to Kafka 2.1.0
- Update to Kafka 2.0.0
- Update to Kafka 1.1.1
- Update to Kafka 1.1.0
- Update to Kafka 1.0.0
- Update to Kafka 0.11.0.0
- Update Akka to 2.5.3
- Added transaction related properties to config options.
- Minor improvements to producer API
- Update Kafka to 0.10.2.1
- Update Akka to 2.4.18
- Update Kafka to 0.10.2.0
- Update dependencies
- Minor tweaks to logging verbosity
- Crossbuild for Scala 2.11.8 and 2.12.1
- Added maxMetaDataAge to KafkaConsumer (thanks @nitendragautam)
- KafkaConsumerActor now terminates on downstream receiver actor termination (thanks @yoks)
- Fixed bug related to Exception Handling in KafkaProducer (thanks @conniec)
- Added new subscription mode: AutoPartitionWithManualOffset.
- Updated Akka to 2.4.14.
- Added new Retry count config, to prevent overwhelming an unresponsive downstream client.
- Supports Kafka 0.10.1.0
- Added support for new Consumer property: max.poll.interval
- Removed log4j-over-slf4j dependency
- Added examples to source
- Subscribe model changed, now supports more explicit subscription types
- Handling of partition rebalances improved
- ConsumerActor failure and restart mechanics improved
- Versioning scheme changed
- Testkit improvements
- ConsumerActor wrapper API provided
- Tested against Kafka 0.10.0.1
- Supports Kafka Client 0.10.0.0
- Add max.poll.records config option to consumer
- Supports Kafka Client 0.9.0.1
YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.
Copyright 2016-2018, Cake Solutions.
Licensed under the MIT License