apache/pulsar-client-node

[Bug] consumer listener does not respect receiver queue size

JTBS opened this issue · 2 comments

JTBS commented

Search before asking

  • I searched in the issues and found nothing similar.

Version

Pulsar Version: 2.11.0
Pulsar C++ Client Version: 3.1.0 (This is version used by Nodejs 1.8.0 but I do see newer C++ version 3.1.1 available)
Pulsar Nodejs Client Version: 1.8.0

We are using Pulsar Nodejs Client that depends on Pulsar C++ client version (details above)

  • When using Consumers with Listener approach, setting "ReceiverQueueSize" is not honored by Consumer
  • Receiver Queue Size works when using Receive approach with Consumer

This issue already addressed in Pulsar Java Client:

Minimal reproduce step

  1. Use Shared subscription that has two below consumers:
  2. Start 1st Consumer with listener callback approach
  3. Start 2nd Consumer with Receive option to receive messages
  4. Set ReceiverQueueSize for both consumers to: 50
  5. Send 1000 messages to topic that Shared Subscription above is subscribed to

For this test Implement Message processing function as below and call the same from both Listener callback and Receive:
ReceiveMessages(msg: any, consumer: any) {
await delay(500); // 500 ms delay
}

Its same method used by Listener Consumer or Receive Consumer.

What did you expect to see?

Listener Consumer: ~500 message to be picked up
Receive Consumer: ~500 messages to be picked up

What did you see instead?

Listener Consumer: You will notice that almost all 1000 or at least more than 900 messages go to Listener
Receive Consumer: Consumer will pick up at most nothing to < 50

Anything else?

In C++ implementation, it looks like listener callbacks are posted as messages are processed from queue using Boost Async IO library. Once caller posts all messages, new messages are retrieved.

If any receive callback does CPU intensive work, there is delay imposed but if we are doing async/await option in NodeJS - example: waiting for DB call to complete, caller assumes we are done processing with all messages in Receiver Queue and request more. This will look like very fast consumer to broker and all messages are pushed, and Listener Consumer will end up taking up a lot before actually getting any finished.

Due to this 2nd Consumer with Receive approach barely get any messages.

This issue is well document and addressed in Pulsar Java Client already but similar fix did not make it C++ client?
apache/pulsar#11008

Is this to be considered as behavior by design for C++ client OR this needs to be flagged as bug and needs fix?
Any thoughts, feedback and responses are appreciated.

Thanks again for your help

Are you willing to submit a PR?

  • I'm willing to submit a PR!
RobertIndie commented

Actually, it's a bug from the nodejs client. I will push a PR to fix it.

JTBS commented

Thank you very much @RobertIndie