psarna/seastar

Implement partitioners

Opened this issue · 0 comments

Implement partitioners: classes which return a partition ID based on: topic, key, value, cluster metadata. The classes can maintain/modify their internal state.

Inspiration from Java client: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/Partitioner.java

Some example partitioners:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java - Sets partition number based on key hash.

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/UniformStickyPartitioner.java - Returns constant partition number for each topic (first it sets it as random)

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/RoundRobinPartitioner.java - Round robin

There could be some caveats, such as number of partitions increasing/decreasing. When reading from metadata (in our project metadata_response) make sure that partition's _error_code is zero. If not, partition can be unavailable (availablePartitions in linked Java files). Possible errors (copied from Java client):

/**
 * Possible topic-level error codes:
 *  UnknownTopic (3)
 *  LeaderNotAvailable (5)
 *  InvalidTopic (17)
 *  TopicAuthorizationFailed (29)

 * Possible partition-level error codes:
 *  LeaderNotAvailable (5)
 *  ReplicaNotAvailable (9)
 */