Implement partitioners
Opened this issue · 0 comments
Implement partitioners: classes which return a partition ID based on: topic, key, value, cluster metadata. The classes can maintain/modify their internal state.
Inspiration from Java client: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/Partitioner.java
Some example partitioners:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java - Sets partition number based on key hash.
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/UniformStickyPartitioner.java - Returns constant partition number for each topic (first it sets it as random)
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/RoundRobinPartitioner.java - Round robin
There could be some caveats, such as number of partitions increasing/decreasing. When reading from metadata (in our project metadata_response
) make sure that partition's _error_code
is zero. If not, partition can be unavailable (availablePartitions
in linked Java files). Possible errors (copied from Java client):
/**
* Possible topic-level error codes:
* UnknownTopic (3)
* LeaderNotAvailable (5)
* InvalidTopic (17)
* TopicAuthorizationFailed (29)
* Possible partition-level error codes:
* LeaderNotAvailable (5)
* ReplicaNotAvailable (9)
*/