Using JSON.NET (Newtonsoft.Json) for serialization and calling client.CreateCheckStatusResponseAsync causes error: System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead

Description

Hello, I am getting this error when using JSON.NET (Newtonsoft.Json) for serialization in Program.cs for my Azure Isolated Durable Function. I followed MS docs for implementation.
https://learn.microsoft.com/en-us/azure/azure-functions/dotnet-isolated-process-guide?tabs=windows#customizing-json-serialization

When I update my Program.cs to use Newtonsoft.json serialization and when httptrigger calls the client.CreateCheckStatusResponseAsync(req, instanceId) method I get following error:


For detailed output, run func with --verbose flag.
[2024-05-14T14:47:15.126Z] Host lock lease acquired by instance ID '000000000000000000000000A4F28EEB'.
[2024-05-14T14:47:50.371Z] Executing 'Functions.DurableFunctionsOrchestrationCSharp_HttpStart' (Reason='This function was programmatically called via the host APIs.', Id=fd30b1b8-057b-436e-a18f-d30814f25a47)
[2024-05-14T14:47:50.624Z] Scheduling new DurableFunctionsOrchestrationCSharp orchestration with instance ID 'a367ec3a68d6471db45eed982eee91a2' and 0 bytes of input data.
[2024-05-14T14:47:50.753Z] Started orchestration with ID = 'a367ec3a68d6471db45eed982eee91a2'.
[2024-05-14T14:47:50.818Z] Executing 'Functions.DurableFunctionsOrchestrationCSharp' (Reason='(null)', Id=7dbebd06-038e-4f26-897d-bd428d126e71)
[2024-05-14T14:47:50.934Z] Function 'DurableFunctionsOrchestrationCSharp_HttpStart', Invocation id 'fd30b1b8-057b-436e-a18f-d30814f25a47': An exception was thrown by the invocation.
[2024-05-14T14:47:50.935Z] Result: Function 'DurableFunctionsOrchestrationCSharp_HttpStart', Invocation id 'fd30b1b8-057b-436e-a18f-d30814f25a47': An exception was thrown by the invocation.
Exception: System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
[2024-05-14T14:47:50.936Z]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
[2024-05-14T14:47:50.937Z]    at System.IO.Stream.Write(ReadOnlySpan`1 buffer)
[2024-05-14T14:47:50.938Z]    at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
[2024-05-14T14:47:50.939Z]    at System.IO.StreamWriter.Dispose(Boolean disposing)
[2024-05-14T14:47:50.939Z]    at System.IO.TextWriter.Dispose()
[2024-05-14T14:47:50.940Z]    at Azure.Core.Serialization.NewtonsoftJsonObjectSerializer.Serialize(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
[2024-05-14T14:47:50.941Z]    at Azure.Core.Serialization.NewtonsoftJsonObjectSerializer.SerializeAsync(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
[2024-05-14T14:47:50.942Z]    at Microsoft.Azure.Functions.Worker.DurableTaskClientExtensions.CreateCheckStatusResponseAsync(DurableTaskClient client, HttpRequestData request, String instanceId, HttpStatusCode statusCode, CancellationToken cancellation) in /_/src/Worker.Extensions.DurableTask/DurableTaskClientExtensions.cs:line 62
[2024-05-14T14:47:50.942Z] Saying hello.
[2024-05-14T14:47:50.943Z]    at FunctionApp4.DurableFunctionsOrchestrationCSharp.HttpStart(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\bernara\source\repos\FunctionApp4\FunctionApp4\Function1.cs:line 52
[2024-05-14T14:47:50.944Z]    at FunctionApp4.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in C:\Users\bernara\source\repos\FunctionApp4\FunctionApp4\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 40
[2024-05-14T14:47:50.945Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-14T14:47:50.946Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 48
[2024-05-14T14:47:50.947Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
Stack:    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
[2024-05-14T14:47:50.948Z]    at System.IO.Stream.Write(ReadOnlySpan`1 buffer)
[2024-05-14T14:47:50.948Z]    at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
[2024-05-14T14:47:50.949Z]    at System.IO.StreamWriter.Dispose(Boolean disposing)
[2024-05-14T14:47:50.950Z]    at System.IO.TextWriter.Dispose()
[2024-05-14T14:47:50.950Z]    at Azure.Core.Serialization.NewtonsoftJsonObjectSerializer.Serialize(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
[2024-05-14T14:47:50.951Z]    at Azure.Core.Serialization.NewtonsoftJsonObjectSerializer.SerializeAsync(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
[2024-05-14T14:47:50.952Z]    at Microsoft.Azure.Functions.Worker.DurableTaskClientExtensions.CreateCheckStatusResponseAsync(DurableTaskClient client, HttpRequestData request, String instanceId, HttpStatusCode statusCode, CancellationToken cancellation) in /_/src/Worker.Extensions.DurableTask/DurableTaskClientExtensions.cs:line 62
[2024-05-14T14:47:50.955Z]    at FunctionApp4.DurableFunctionsOrchestrationCSharp.HttpStart(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\bernara\source\repos\FunctionApp4\FunctionApp4\Function1.cs:line 52
[2024-05-14T14:47:50.955Z]    at FunctionApp4.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in C:\Users\bernara\source\repos\FunctionApp4\FunctionApp4\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 40
[2024-05-14T14:47:50.956Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-14T14:47:50.957Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 48
[2024-05-14T14:47:50.958Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77.
[2024-05-14T14:47:50.987Z] Executed 'Functions.DurableFunctionsOrchestrationCSharp_HttpStart' (Failed, Id=fd30b1b8-057b-436e-a18f-d30814f25a47, Duration=640ms)
[2024-05-14T14:47:50.988Z] Executed 'Functions.DurableFunctionsOrchestrationCSharp' (Succeeded, Id=7dbebd06-038e-4f26-897d-bd428d126e71, Duration=185ms)
[2024-05-14T14:47:50.989Z] System.Private.CoreLib: Exception while executing function: Functions.DurableFunctionsOrchestrationCSharp_HttpStart. System.Private.CoreLib: Result: Failure
Exception: System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
[2024-05-14T14:47:50.991Z]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
[2024-05-14T14:47:50.992Z]    at System.IO.Stream.Write(ReadOnlySpan`1 buffer)
[2024-05-14T14:47:50.993Z]    at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
[2024-05-14T14:47:50.993Z]    at System.IO.StreamWriter.Dispose(Boolean disposing)
[2024-05-14T14:47:50.994Z]    at System.IO.TextWriter.Dispose()
[2024-05-14T14:47:50.995Z]    at Azure.Core.Serialization.NewtonsoftJsonObjectSerializer.Serialize(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
[2024-05-14T14:47:50.996Z]    at Azure.Core.Serialization.NewtonsoftJsonObjectSerializer.SerializeAsync(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
[2024-05-14T14:47:50.996Z]    at Microsoft.Azure.Functions.Worker.DurableTaskClientExtensions.CreateCheckStatusResponseAsync(DurableTaskClient client, HttpRequestData request, String instanceId, HttpStatusCode statusCode, CancellationToken cancellation) in /_/src/Worker.Extensions.DurableTask/DurableTaskClientExtensions.cs:line 62
[2024-05-14T14:47:50.997Z]    at FunctionApp4.DurableFunctionsOrchestrationCSharp.HttpStart(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\bernara\source\repos\FunctionApp4\FunctionApp4\Function1.cs:line 52
[2024-05-14T14:47:50.998Z]    at FunctionApp4.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in C:\Users\bernara\source\repos\FunctionApp4\FunctionApp4\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 40
[2024-05-14T14:47:50.999Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-14T14:47:51.001Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 48
[2024-05-14T14:47:51.002Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
[2024-05-14T14:47:51.003Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack:    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
[2024-05-14T14:47:51.003Z]    at System.IO.Stream.Write(ReadOnlySpan`1 buffer)
[2024-05-14T14:47:51.004Z]    at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
[2024-05-14T14:47:51.004Z]    at System.IO.StreamWriter.Dispose(Boolean disposing)
[2024-05-14T14:47:51.005Z]    at System.IO.TextWriter.Dispose()
[2024-05-14T14:47:51.005Z]    at Azure.Core.Serialization.NewtonsoftJsonObjectSerializer.Serialize(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
[2024-05-14T14:47:51.005Z]    at Azure.Core.Serialization.NewtonsoftJsonObjectSerializer.SerializeAsync(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
[2024-05-14T14:47:51.006Z]    at Microsoft.Azure.Functions.Worker.DurableTaskClientExtensions.CreateCheckStatusResponseAsync(DurableTaskClient client, HttpRequestData request, String instanceId, HttpStatusCode statusCode, CancellationToken cancellation) in /_/src/Worker.Extensions.DurableTask/DurableTaskClientExtensions.cs:line 62
[2024-05-14T14:47:51.007Z]    at FunctionApp4.DurableFunctionsOrchestrationCSharp.HttpStart(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\bernara\source\repos\FunctionApp4\FunctionApp4\Function1.cs:line 52
[2024-05-14T14:47:51.008Z]    at FunctionApp4.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in C:\Users\bernara\source\repos\FunctionApp4\FunctionApp4\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 40
[2024-05-14T14:47:51.009Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-14T14:47:51.009Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 48
[2024-05-14T14:47:51.010Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
[2024-05-14T14:47:51.011Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88.
[2024-05-14T14:47:51.035Z] Executing 'Functions.SayHello' (Reason='(null)', Id=6df56235-c0df-4362-bcd6-1023db5ad706)
[2024-05-14T14:47:51.044Z] Saying hello to Tokyo.
[2024-05-14T14:47:51.047Z] Executed 'Functions.SayHello' (Succeeded, Id=6df56235-c0df-4362-bcd6-1023db5ad706, Duration=15ms)
[2024-05-14T14:47:51.072Z] Executing 'Functions.DurableFunctionsOrchestrationCSharp' (Reason='(null)', Id=caccce80-a673-48a7-a7a1-d996b6acd4f4)
[2024-05-14T14:47:51.082Z] Executed 'Functions.DurableFunctionsOrchestrationCSharp' (Succeeded, Id=caccce80-a673-48a7-a7a1-d996b6acd4f4, Duration=11ms)
[2024-05-14T14:47:51.090Z] Executing 'Functions.SayHello' (Reason='(null)', Id=7bf8e96d-0714-4c07-bac6-81f181df8787)
[2024-05-14T14:47:51.092Z] Saying hello to Seattle.
[2024-05-14T14:47:51.095Z] Executed 'Functions.SayHello' (Succeeded, Id=7bf8e96d-0714-4c07-bac6-81f181df8787, Duration=5ms)
[2024-05-14T14:47:51.104Z] Executing 'Functions.DurableFunctionsOrchestrationCSharp' (Reason='(null)', Id=fb60d86e-f41e-4226-b159-6b675077c8cd)
[2024-05-14T14:47:51.109Z] Executed 'Functions.DurableFunctionsOrchestrationCSharp' (Succeeded, Id=fb60d86e-f41e-4226-b159-6b675077c8cd, Duration=5ms)
[2024-05-14T14:47:51.116Z] Executing 'Functions.SayHello' (Reason='(null)', Id=2d4c851a-bed2-4e80-bd8c-7ef0515555a3)
[2024-05-14T14:47:51.118Z] Saying hello to London.
[2024-05-14T14:47:51.120Z] Executed 'Functions.SayHello' (Succeeded, Id=2d4c851a-bed2-4e80-bd8c-7ef0515555a3, Duration=4ms)
[2024-05-14T14:47:51.128Z] Executing 'Functions.DurableFunctionsOrchestrationCSharp' (Reason='(null)', Id=4a74c54b-849b-4aa3-90cc-d36194288d24)
[2024-05-14T14:47:51.137Z] Executed 'Functions.DurableFunctionsOrchestrationCSharp' (Succeeded, Id=4a74c54b-849b-4aa3-90cc-d36194288d24, Duration=9ms)

Steps to reproduce

Update Program.cs file to include serialization to JSON.NET (Newtonsoft.Json)
https://learn.microsoft.com/en-us/azure/azure-functions/dotnet-isolated-process-guide?tabs=windows#customizing-json-serialization

var host = new HostBuilder()
       .ConfigureFunctionsWebApplication((IFunctionsWorkerApplicationBuilder builder) =>
       {
           builder.Services.Configure<WorkerOptions>(workerOptions =>
           {
               var settings = NewtonsoftJsonObjectSerializer.CreateJsonSerializerSettings();
               settings.ContractResolver = new CamelCasePropertyNamesContractResolver();
               settings.NullValueHandling = NullValueHandling.Ignore;

               workerOptions.Serializer = new NewtonsoftJsonObjectSerializer(settings);

           });
       })
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();

Run sample Durable Function and observed error when client.CreateCheckStatusResponseAsync() is invoked.
Sample code taken from: https://github.com/Azure/azure-functions-templates/blob/dev/Functions.Templates/Templates/DurableFunctionsOrchestration-CSharp-Isolated/DurableFunctionsOrchestrationCSharp.cs

  public static class DurableFunctionsOrchestrationCSharp
  {
      [Function(nameof(DurableFunctionsOrchestrationCSharp))]
      public static async Task<List<string>> RunOrchestrator(
          [OrchestrationTrigger] TaskOrchestrationContext context)
      {
          ILogger logger = context.CreateReplaySafeLogger(nameof(DurableFunctionsOrchestrationCSharp));
          logger.LogInformation("Saying hello.");
          var outputs = new List<string>();

          // Replace name and input with values relevant for your Durable Functions Activity
          outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "Tokyo"));
          outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "Seattle"));
          outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "London"));

          // returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
          return outputs;
      }

      [Function(nameof(SayHello))]
      public static string SayHello([ActivityTrigger] string name, FunctionContext executionContext)
      {
          ILogger logger = executionContext.GetLogger("SayHello");
          logger.LogInformation("Saying hello to {name}.", name);
          return $"Hello {name}!";
      }

      [Function("DurableFunctionsOrchestrationCSharp_HttpStart")]
      public static async Task<HttpResponseData> HttpStart(
          [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
          [DurableClient] DurableTaskClient client,
          FunctionContext executionContext)
      {
          ILogger logger = executionContext.GetLogger("DurableFunctionsOrchestrationCSharp_HttpStart");

          // Function input comes from the request content.
          string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
              nameof(DurableFunctionsOrchestrationCSharp));

          logger.LogInformation("Started orchestration with ID = '{instanceId}'.", instanceId);

          // Returns an HTTP 202 response with an instance management payload.
          // See https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-http-api#start-orchestration
          return await client.CreateCheckStatusResponseAsync(req, instanceId);
      }
  }

Hi @ainsleybernard, this is happening due to a known bug with using ConfigureFunctionsWebApplication where the serializer configured via worker options is not being properly registered.

In the meantime, you should be able to configure the Newtonsoft serializer using ConfigureFunctionsWorkerDefaults (you will need to remove the Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore pkg).

Please follow this issue for updates on this bug:

#2131

Actually, going to reopen this and investigate this specific error message you're seeing. It's definitely related to the AspNetCore changes but the error looks different to the others. Will let you know if I have any updates.

After further investigation, I am finding that the serializer is correctly being set and used so that is not the real issue here. There is something else going on with the AspNetCore extension and the output binding.

I found this which is helpful:

dotnet/aspnetcore#7644

The JSON serializer is synchronously writing to the response body.

The workaround I saw was to enable this via configuration:

services.Configure<KestrelServerOptions>(options =>
{
    options.AllowSynchronousIO = true;
});

// Or if using IIS:
services.Configure<IISServerOptions>(options =>
{
    options.AllowSynchronousIO = true;
});

However, when I tried this myself I found a new error at the stage where the worker tries to convert the response to an RPC message to send to the host:

azure-functions-dotnet-worker/src/DotNetWorker.Grpc/RpcExtensions.cs

Line 90 in b0e7178

StatusCode = ((int)response.StatusCode).ToString()

{System.ObjectDisposedException: IFeatureCollection has been disposed.
Object name: 'Collection'.
   at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed()
   at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ContextDisposed()
   at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.Fetch[TFeature](TFeature& cached, Func`2 factory)
   at Microsoft.AspNetCore.Http.DefaultHttpResponse.get_StatusCode()
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.AspNetCoreHttpResponseData.get_StatusCode() in /Users/likasem/source/functions/azure-functions-dotnet-worker/extensions/Worker.Extensions.Http.AspNetCore/src/HttpDataModel/AspNetCoreHttpResponseData.cs:line 39
   at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToRpcHttpAsync(HttpResponseData response, ObjectSerializer serializer) in /Users/likasem/source/functions/azure-functions-dotnet-worker/src/DotNetWorker.Grpc/RpcExtensions.cs:line 88
   at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToRpcAsync(Object value, ObjectSerializer serializer) in /Users/likasem/source/functions/azure-functions-dotnet-worker/src/DotNetWorker.Grpc/RpcExtensions.cs:line 35
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in /Users/likasem/source/functions/azure-functions-dotnet-worker/src/DotNetWorker.Grpc/Handlers/InvocationHandler.cs:line 110}

Will keep digging but wanted to share a quick update.

@liliankasem Thanks for looking into this issue! I will use ConfigureFunctionsWorkerDefaults in the meantime.

I would say this is a problem with the Newtonsoft.Json serializer and not functions nor durable's fault.

CreateCheckStatusResponseAsync uses async apis to write: https://github.com/Azure/azure-functions-durable-extension/blob/f4aed61fde568769519e42242ed3e32c63a0a328/src/Worker.Extensions.DurableTask/DurableTaskClientExtensions.cs#L68. If Newtonsoft.Json performs synchronous serialization within the async call, then that is an issue with the serializer. You will need to configure AllowSynchronousIO as @liliankasem mentioned.