confluentinc/confluent-kafka-python

Expose partitioners to python

pavelschon opened this issue · 0 comments

Description

We have use case where we need to manually calculate partition ID using the murmur2 partitioner from the librdkafka.
Rationale: we need to verify that messages received belongs to the right partition.

Since the desired function is not exposed to python via confluent_kafka, we have a workaround with using ctypes, see below.
But we still think the right approach is to use desired function (if available) via confluent_kafka module.

How to reproduce

librdkafka = ctypes.cdll.LoadLibrary("librdkafka.so.1")

assert librdkafka.rd_kafka_msg_partitioner_murmur2(None, key, len(key), num_partitions, None, None) == expected_partition

Checklist

Please provide the following information:

  • confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 2.3.0
  • Apache Kafka broker version: ?
  • Operating system: -
  • Provide client logs (with 'debug': '..' as necessary) -
  • Provide broker log excerpts -
  • Critical issue -