Redis-Based State Stores for Kafka Streams
Kafka Streams provides in-memory state stores and disk-based state stores (based on RocksDB) out of the box. While these options work great in most scenarios, they both rely on local storage and are not the best fit for continoulsy delivered apps on immutable servers (or stateless containers). In those scenarios, using an external state store (such as Redis) allows to save the application state when replacing the whole machine or the application container. This can allow smoother delivery, as the state stores do not have to be rebuilt before the app can start processing records.
repositories {
jcenter()
}
dependencies {
compile group: 'com.github.andreas-schroeder', name: 'redisks_2.12', version: '0.0.0'
}
resolvers += Resolver.bintrayRepo("and-schroeder", "maven")
libraryDependencies += "com.github.andreas-schroeder" %% "redisks" % "0.0.0"
Redisks depends on lettuce for connecting to Redis. A KeyValueStore (to be used to store the state of a KTable, e.g.), can be created as follows
import com.lambdaworks.redis.RedisClient;
import com.github.andreas_schroeder.redisks.RedisStore;
RedisClient client = ...
StateStoreSupplier<KeyValueStore> store = RedisStore.<String, String>keyValueStore(name)
.withClient(client)
.withKeys(Serdes.String())
.withValues(Serdes.String())
.withKeyComparator(Comparator.naturalOrder())
.build();
Some examples for using the Redis state stores can be found in the acceptance tests.
The benchmark can be found here. The results below were obtained on a MacBook Pro (Retina, 15-inch, Mid 2015), with a 2,8 GHz Intel Core i7 CPU. Currently, the attainable throughput depends significantly on the size of the key-value entries stored.
key/value size | put throughput | get throughput |
---|---|---|
100 bytes | 6.04 MiB/Sec | 2.23 MiB/Sec |
1024 bytes | 57.28 MiB/Sec | 19.43 MiB/Sec |
2048 bytes | 94.58 MiB/Sec | 33.13 MiB/Sec |
4000 bytes | 129.53 MiB/Sec | 50.79 MiB/Sec |