Expose partitioners to python
pavelschon opened this issue · 0 comments
pavelschon commented
Description
We have use case where we need to manually calculate partition ID using the murmur2 partitioner from the librdkafka.
Rationale: we need to verify that messages received belongs to the right partition.
Since the desired function is not exposed to python via confluent_kafka
, we have a workaround with using ctypes
, see below.
But we still think the right approach is to use desired function (if available) via confluent_kafka
module.
How to reproduce
librdkafka = ctypes.cdll.LoadLibrary("librdkafka.so.1")
assert librdkafka.rd_kafka_msg_partitioner_murmur2(None, key, len(key), num_partitions, None, None) == expected_partition
Checklist
Please provide the following information:
- confluent-kafka-python and librdkafka version (
confluent_kafka.version()
andconfluent_kafka.libversion()
): 2.3.0 - Apache Kafka broker version: ?
- Operating system: -
- Provide client logs (with
'debug': '..'
as necessary) - - Provide broker log excerpts -
- Critical issue -