HighLevelProducer with KeyedPartitioner fails on first send
itamarwe opened this issue · 9 comments
When using KeyedParitioner with the HighLevelProducer the first send fails with
BrokerNotAvailableError: Could not find the leader
Consecutive sends work perfectly.
Yeah. I'm getting the same error no matter what I do.
The random partitioner and default parttitioners work. random and keyed both fail with the BrokerNotAvailableError.
Only for the first message, though. (or until the topic metadata is updated?)
I've created a pull-request to fix it:
#378
As suspected, the metadata isn't updated when the producer is initialized, so the partitions array is empty, which causes the keyed partitioner to return NaN as the chosen partitions, which fails the producer.
Yup, same error. I have also found the documentation for working with keyed partitions very poor (probably because they don't work). I think this is something that could be improved upon as well.
I just ended up rolling my own and manually setting partition number in payloads.
I ran into this problem today, and I was able to get around it by calling client.loadMetadataForTopics([topic]) and then calling client.updateMetadatas(metadata) in the callback function. After calling that, I called back my own callback saying that the producer is ready to use.
I did all this with several promises, so it might be more difficult to do with just using callbacks. Basically, I didn't resolve the "producer" promise until after updating the metadata for the topic on the client.
I've attemped to create a minimal example to reproduce this issue in this gist. I say attempted, because there's no error thrown with v4.0.1. I wasn't able to install v0.3.2, which seems to be what was available back then (and I'm not super keen on degrading my Arch system back to dinosaur). Is anyone able to use the example to confirm that this used to be an issue and isn't anymore?