Azure/azure-functions-durable-extension

RpcException "ResourceExhausted" while attempting to read orchestrator output if the input is greater than 4MB

DCorrBECare opened this issue · 7 comments

DCorrBECare commented

Description

I'm scheduling an orchestrator with an input bigger than 4MB (which is now possible thanks to #2778). Everything works as expected except when i try to read the orchestrator's output.

Expected behavior

I can read the orchestrator's output without an exception being thrown.

Actual behavior

Exception: Grpc.Core.RpcException: Status(StatusCode="ResourceExhausted", Detail="Received message exceeds the maximum configured message size.")

Relevant source code snippets

FunctionApp1.zip

Known workarounds

No known workarounds

App Details

  • Microsoft.Azure.Functions.Worker.Extensions.DurableTask: 1.1.3
  • Azure Functions version: v4
  • Programming language used: C# net8.0

Screenshots

OrchestratorBug.mp4
DCorrBECare commented

Hi team, any update on this?

davidmrdavid commented

Sorry for the delay here. This seems to be an oversight in the GetInstances API, which is probably making a GRPC call with some default size limit somewhere.

cc/ @jviau in case he's able to easily find the source code where this config can be specified. If this is an easy fix, there's no reason why we couldn't sneak in that fix in the coming weeks.

@DCorrBECare: are you able to work around this issue by not passing this "large" data directly, instead utilizing a claim-check pattern (i.e passing a lightweight identifier that you can use to download the data when needed)? Just thinking of workarounds as it may take us a bit to prioritize this, especially given that it is a DF anti-pattern to directly deal with inputs that are MBs large.

jviau commented

@DCorrBECare can you provide a minimal repro?

  • Where are you reading the orchestrators output?
  • How large is this output?

Even though we set the max message length to int.MaxValue, this doesn't mean you can use inputs/outputs of arbitrary size. These are still network calls that have constraints out of our control applied to them.

DCorrBECare commented

Hi @jviau @davidmrdavid, thank you for your feedback.

Yes I am able to workaround this issue with a claim-check pattern, that's what i have been doing till this time.

I am reading the output of the orchestrator inside an HTTP function, the output size doesn't really matter if the input is greater than 4MB. In the repro I attached in the issue the output is only one char.

kemmis commented

We started having this issue recently.. possibly after upgrading to .net 8 or newer function nuget packages. In the past we were able to pass large payloads and it worked fine. The issue started popping up today. Any update on work to get this fixed?

kemmis commented

Looking back at our logs, it seems like we have been getting this ResourceExhausted error since 5/15/2024.