Azure/azure-functions-eventhubs-extension

The lease ID specified did not match the lease ID for the blob occurs

Closed this issue · 4 comments

#### Actual behavior
It is configured to receive and process messages detected from ServiceBus with Function in Azure environment.
When I checked Application Insights, I found a Microsoft.WindowsAzure.Storage.StorageException.
I want to know the cause of this phenomenon

A similar issue is registered, but it has not been resolved.
"The lease ID specified did not match the lease ID for the blob crashes worker."
Azure/azure-webjobs-sdk#2103

#### Repro steps
The reproduction procedure is not known because it has occurred only once.

Application Insights logs

Message:
The lease ID specified did not match the lease ID for the blob. The lease ID specified did not match the lease ID for the blob.

Exception type:
Microsoft.WindowsAzure.Storage.StorageException

Failed method:
Microsoft.WindowsAzure.Storage.Core.Executor.Executor+d__4`1.MoveNext

Call stack:

Microsoft.Azure.EventHubs.Processor.LeaseLostException:
   at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<RenewLeaseCoreAsync>d__34.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<UpdateLeaseCoreAsync>d__38.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<UpdateCheckpointAsync>d__18.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.EventHubs.Processor.PartitionContext+<PersistCheckpointAsync>d__40.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.EventHubs.EventHubListener+EventProcessor+<Microsoft-Azure-WebJobs-EventHubs-EventHubListener-ICheckpointer-CheckpointAsync>d__16.MoveNext (Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.EventHubs\Listeners\EventHubListener.csMicrosoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 217)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.EventHubs.EventHubListener+EventProcessor+<CheckpointAsync>d__13.MoveNext (Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.EventHubs\Listeners\EventHubListener.csMicrosoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 184)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.EventHubs.EventHubListener+EventProcessor+<ProcessEventsAsync>d__12.MoveNext (Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.EventHubs\Listeners\EventHubListener.csMicrosoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 176)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.EventHubs.Processor.PartitionPump+<ProcessEventsAsync>d__30.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<RenewLeaseCoreAsync>d__34.MoveNext で処理された内部例外 Microsoft.WindowsAzure.Storage.StorageException:
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor+<ExecuteAsyncInternal>d__4`1.MoveNext (Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<RenewLeaseCoreAsync>d__34.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)

#### Package version
Microsoft.Azure.WebJobs.ServiceBus 3.0.0-beta8
.NET Standard 2.0

The issue you linked to is unrelated (although they're both related to storage) -- this one should not be crashing the worker. If it is, please let me know. It appears that this is happening in EventHubs, which means that it's likely that this is completely normal, see here: https://stackoverflow.com/a/41867611/3516125.

We do have an issue to hide some of these transient errors (or at least make them warnings): #3

I'm going to close this as this error is expected. But we may be able to clean it up via the linked issue.

alrod commented

@shibuya-takahiro, also Microsoft.Azure.WebJobs.ServiceBus 3.0.0-beta8 package is pretty old.
try to use new one:
https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.ServiceBus/3.0.6
Most likely it was fixed here:
8380ea2f

@brettsam , @alrod
Thank you for the information.

also Microsoft.Azure.WebJobs.ServiceBus 3.0.0-beta8 package is pretty old. try to use new one:

I will try the new version.