RDKafka producer publishes messages but doesn't reflect in message count

[sampather]

Hi Team,

Am using rdkafka version 0.9.1 with .net core.
When i deploy it on ubuntu server 14.X OS the publisher posts message and can be seen with the increase in message.
The same project is being deployed on windows server 2012 R2 standard, in this publisher doesn't error out it says the publish was successful but doesn't reflect the same in message counts.
But when deployed on windows 7 sp1 the publish is success and reflects in message count.

Can anyone tell whats the problem here. am i missing anything.

I do have a strange problem with ubuntu, when i try and load test with like 1000 messages , the cpu consumption raises , it's just that publisher.Dispose() doesn't respond at all and causes the system to crash

[treziac]

Hi @sampather

Do the sample programs work well? If it does, could you share your code?
Also note a new version of librdkafka was released last week and fixed some bug, you might want to try the latest nuget prerelease of rdkafka to check if it corrects the bug (https://github.com/edenhill/librdkafka/releases/tag/v0.9.2)

[sampather]

Hi @treziac

Sample with disposing producer also doesn't work for me.

Below is the code snippet i use for publishing message:

List tasks = new List();
using (Topic myTopic = _rdProducer.Topic(topic))
{
foreach (var messagesToSendWithKey in messagesToSendWithKeys)
{
byte[] data = Encoding.UTF8.GetBytes(messagesToSendWithKey.Value);
byte[] key = Encoding.UTF8.GetBytes(messagesToSendWithKey.Key);
tasks.Add(myTopic.Produce(data, key));
}
}

        Task.WaitAll(tasks.ToArray(), TimeSpan.FromSeconds(4));

[treziac]

There were sometimes issues while disposing in 0.9.1 librdkafka (for example if you have a group.id in your producer config), try switching to latest prerelease to see if you still reproduce.

For you windows server case, What do you mean exactly by "publish was successful but doesn't reflect the same in message counts" ? No message receive on broker? You may have connectivity issue in this case, check your broker logs and if nothing is relevant here, enable rdkafka logs (config["debug"] = "broker,topic" for example)

Note that you don't have to wait for produce - when you dispose your producer, your queued request will wait to be acked before disposing, you should still checked they completed successfully