auth0/auth0.net

ImportUsersAsync() fails on retry

LBognanni opened this issue · 4 comments

LBognanni commented

Sometimes, when we import users via ImportUsersAsync(), this operation fails with System.InvalidOperationException: The stream was already consumed. It cannot be read again.
I think this is because SendAsync() in HttpClientManagementConnection will retry the operation when we get rate-limited.

However, I suspect that because the file parameter of ImportUsersAsync() is a stream and has already been read once (and maybe disposed?), the above exception gets raised.

Environment

  • Version of this library used: 7.17.4
  • Which framework are you using, if applicable: net6.0
  • Other modules/plugins/libraries that might be involved:
  • Any other relevant information you think would be useful:
LBognanni commented

Relevant portion of the stack trace:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.InvalidOperationException: The stream was already consumed. It cannot be read again.
   at System.Net.Http.StreamContent.PrepareContent()
   at System.Net.Http.StreamContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.InternalCopyToAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.MultipartContent.SerializeToStreamAsyncCore(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Datadog.Trace.ClrProfiler.CallTarget.Handlers.Continuations.TaskContinuationGenerator`4.SyncCallbackHandler.ContinuationAction(Task`1 previousTask, TTarget target, CallTargetState state)
   at Datadog.Trace.ClrProfiler.CallTarget.Handlers.Continuations.TaskContinuationGenerator`4.SyncCallbackHandler.ContinuationAction(Task`1 previousTask, TTarget target, CallTargetState state)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendCoreAsync(HttpRequestMessage request, Context context, CancellationToken cancellationToken)
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Auth0.ManagementApi.HttpClientManagementConnection.SendRequest[T](HttpRequestMessage request, JsonConverter[] converters, CancellationToken cancellationToken)
   at Auth0.ManagementApi.HttpClientManagementConnection.SendAsyncInternal[T](HttpMethod method, Uri uri, Object body, IDictionary`2 headers, IList`1 files, JsonConverter[] converters, CancellationToken cancellationToken)
   at Auth0.ManagementApi.HttpClientManagementConnection.<>c__DisplayClass14_0`1.<<SendAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Auth0.ManagementApi.HttpClientManagementConnection.Retry[TResult](Func`1 retryable)
   at Auth0.ManagementApi.HttpClientManagementConnection.SendAsync[T](HttpMethod method, Uri uri, Object body, IDictionary`2 headers, IList`1 files, JsonConverter[] converters, CancellationToken cancellationToken)
frederikprijck commented

Thanks for reporting. Will need to look into this.

For the time being, would it be possible for you to catch the error yourself and retry with a new stream?

  • 👍1
frederikprijck commented

Having debugged this, I wanted to give an update on what's happening for future reference.

When the using statement is exited, the HttpRequestMessage is disposed, which disposes the Content, which on its turn Disposes the stream, see:

Long story short, it looks like it's expected and we might need to consider changing the way how we do the RateLimit retries.

Will need to continue looking into it, but again, for the time being I would recommend anyone to catch the Stream error themselves and recreate the stream and try again.

frederikprijck commented

As elaborated above, this is originated internally in the framework and hard to solve.

Closing this for now, I would recommend to catch the error and handle it yourself.