aws/aws-xray-sdk-node

captureAWSClient adding significant latency to lambda invocations

Opened this issue · 4 comments

When testing the durations of our lambda invocations we are noticing a difference of close to a second in duration time when the lambda clients are wrapped with captureAWSClient. We are using version 3.3.1 of aws-xray-sdk-core

Hi @JohnDupuis
I'm trying to understand your application architecture. Do you have 2 lambda functions, with Lambda1 invoking Lambda2 via an AWS Lambda client? And it is the Lambda1 invocation duration that is increasing by ~1 second when using the lambda client instrumentation?

Can you provide some more details like code snippet for client instrumentation, debug logs from the AWS X-Ray SDK, etc?

Thanks!

Similar symptom here, but not sure it's the same root cause:

We have noticed our lambda execution times are slowed down significantly when we use captureAWSClient. Tracking down the problem, we have isolated it to the X-Ray SDK attempting to capture a stack trace upfront of any AWS client request. Since we have compiled TypeScript code with source maps enabled, capturing the stack trace triggers source maps parsing and resolution, which is extremely expensive, especially that the traces are only useful in case an exception is actually thrown, which is an extremely rare occurrence.

After we have changed the following line in the X-Ray SDK from:

  var stack = (new Error()).stack;

to

  var stack = "";

our lambda execution time went from >18sec to <6sec.

We understand this is a workaround for the well known node issue that truncates the stack traces to the event loop ticker but perhaps there is a way to make it optional, or find a way to get the stack trace cheaper, and resolve it with the source maps only in case there is an error.

Here's the offending line:

var stack = (new Error()).stack;

any updates on it?

I had exactly the same issue and @georgeionita solution helped.