Azure/azure-sdk-for-python

ServiceBusReceiver receive_messages max_wait_time race condition locks the message but sends back empty list

cabal-daniel opened this issue · 5 comments

cabal-daniel commented

  • Package Name: pip install azure-servicebus

  • Package Version: 7.12.2

  • Operating System: MacOS and Ubuntu

  • Python Version: 3.8.18

Describe the bug

Receiving code:

with sb_client.get_queue_receiver(queue) as receiver:
    messages = receiver.receive_messages(
        max_message_count=max_message_count,
        max_wait_time=max_wait_time,
    )
    print(messages)
    for message in messages:
           receiver.complete_message(message)

If max_wait_time is set in ServiceBusReceiver.receive_messages, then a race condition can happen where a message gets locked by the receive_message action but receive_messages returns back an empty list, locking this message for the duration of the lockout period.

To Reproduce
Steps to reproduce the behavior:

  1. We used max_wait_time to 2 and are receiving 1 message at a time
  2. Spin up 10 workers, constantly calling the Receiving code above in a polling interval in a while loop
  3. Have a sender send 100 or so messages to the queue 1 at a time with a decently large body
    Actual behaviour: some of the messages will get locked in the queue. From our experiments roughly 2-4 messages out of 100 get put into the locked state without them being complete. The receive_messages method shouldn't lock a message unless it can return that message in full.

Expected behavior
All the messages will get completed. No message would get locked out.

I suspect the code for that behaviour is somewhere in here: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/servicebus/azure-servicebus/azure/servicebus/_servicebus_receiver.py#L454

Perhaps it can be configured so that max_time_out can allow a bit of "grace period" to allow the full message to be received and then terminate afterwards once the max_time_out is reached.

github-actions commented

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @EldertGrootenboer.

kashifkhan commented

Thank you for the feedback @cabal-daniel . We will investigate and get back to you asap.

  • 👍1