loekd/ServiceFabric.ServiceBus

Lock is Expired Error on NetStandardPackage

Closed this issue · 7 comments

It looks like there are multiple things attempting to Complete a queue message at once and therefore a message about the Lock being expired or already closed is coming.

loekd commented

Can you please add some more info about how to reproduce this issue?
Does your IServiceBusMessageReceiver have Receiver.AutoComplete set to true while you are completing the message in your own code as well?

hi @loekd, we are also running into this issue with the NetStandardPackage where our service bus is constantly throwing this message: There was an error while receiving a message: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue.

I converted your non-NetStandard demo into what the NetStandard version so the implementation is very similar.

Here is how CreateServiceInstanceListeners for the stateless service looks like:

 yield return new ServiceInstanceListener(context => new ServiceBusQueueCommunicationListener(
    cl => new WorkerMessageHandler(cl, logAction, "StatelessService-ServiceBusQueueListener")
    {
        AutoComplete = false
    }
    , context
    , serviceBusQueueName
    , serviceBusSendConnectionString: null
    , serviceBusReceiveConnectionString: connectionString)
{
    AutoRenewTimeout = TimeSpan.FromSeconds(70),  //auto renew up until 70s, so processing can take no longer than 60s (default lock duration).
    LogAction = logAction,
    MessagePrefetchCount = 10
}, "StatelessService-ServiceBusQueueListener");

And here is the handler:

protected override Task ReceiveMessageImplAsync(Message message, CancellationToken cancellationToken)
{
    string msg = Encoding.UTF8.GetString(message.Body);

    WriteLog($"{msg} -- Sleeping while processing queue message {message.MessageId} to test message lock renew function (send more than 9 messages!).");

    Thread.Sleep(TimeSpan.FromSeconds(5));

    WriteLog($"Handling queue message {message.MessageId}");
    return Task.FromResult(true);
}

Running this with a single instance of the app will process some messages and throw the lock expired message for others. Is each message being completed twice? The AutoComplete is set to false.

loekd commented

I found an issue that completed messages twice if AutoComplete is true.
This explains the original incorrect behavior reported by @gdodd1977 , but not your issue @hakkio .

In your case, could the issue be that you're not completing the message at all now, causing it to be received over and over again, until the lock times out?

With AutoComplete set to false, how would we complete it manually? I see there is a Complete method under the Listener for DefaultServiceBusMessageReceiver. But when I call it I get:

"There was an error while receiving a message: Operation is not valid due to the current state of the object."

loekd commented

So if you set AutoComplete to true it works?

There's a Complete method on DefaultServiceBusMessageReceiver as well now.
It's called automatically when AutoComplete is true:

await ReceiveMessageImplAsync(message, cancellationToken).ConfigureAwait(false);
if (AutoComplete)
{
     await CompleteMessage(message).ConfigureAwait(false);
}

With autocomplete set to True, i was getting multiple Completes. With it off, it wasn't completing at all. Ended up implementing a different receiver which works. Not sure what the issue was with your package.

loekd commented

I believe this was fixed on may 17th.