DataDog/datadog-lambda-js

Using sendDistributionMetricWithDate gives runtime error on flush

timotheeg opened this issue · 2 comments

timotheeg commented

Expected Behavior

Lambda using sendDistributionMetricWithDate() should work and metrics should be sent

Actual Behavior

In my Lambda, I have 2 calls to send metrics. I initially used sendDistributionMetric(), but decided to provide both the same date with sendDistributionMetricWithDate() instead. The metrics are accepted, I can see the log entries:

{
    "e": 1728552052.824,
    "m": "cx.helpdesk.call.rating",
    "t": [
        "dd_lambda_layer:datadog-nodev20.17.0",
        "region:ap-southeast-1",
        "memorysize:128",
        "cold_start:true",
        "datadog_lambda:v9.115.0",
        "runtime:nodejs20.x",
        // ... plus private tags
    ],
    "v": 6
}

(why is this even printed in the log btw? did someone forgot to remove a console log?)

But at the end of processing, this error is logged:

2024-10-10T09:20:53.845Z	76aa8b68-d34d-4561-8181-faa7b0cd8cf9	ERROR	[dd.trace_id=3371094672245296109 dd.span_id=7681230763497218352] 
{
    "status": "error",
    "message": "Cannot read properties of undefined (reading 'flush')",
    "name": "TypeError",
    "stack": "TypeError: Cannot read properties of undefined (reading 'flush')
    at MetricsListener.<anonymous> (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:134:56)
    at step (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:33:23)
    at Object.next (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:14:53)
    at fulfilled (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:5:58)"
}

Steps to Reproduce the Problem

  1. Make a lambda in ap-southeast-1
  2. Use the 2 DD layers:
    • arn:aws:lambda:ap-southeast-1:464622532012:layer:Datadog-Node20-x:115
    • arn:aws:lambda:ap-southeast-1:464622532012:layer:Datadog-Extension:65
  3. set up the usual DD env vars to retrieve API key and everything
  4. use sendDistributionMetric() for 2 metrics, and verify data appears in DD
  5. change to use sendDistributionMetricWithDate() for the 2 metrics, and see the runtime errors in cloudwatch logs

Specifications

* `arn:aws:lambda:ap-southeast-1:464622532012:layer:Datadog-Node20-x:115`
* `arn:aws:lambda:ap-southeast-1:464622532012:layer:Datadog-Extension:65`

Stacktrace

2024-10-10T09:20:53.845Z	76aa8b68-d34d-4561-8181-faa7b0cd8cf9	ERROR	[dd.trace_id=3371094672245296109 dd.span_id=7681230763497218352] 
{
    "status": "error",
    "message": "Cannot read properties of undefined (reading 'flush')",
    "name": "TypeError",
    "stack": "TypeError: Cannot read properties of undefined (reading 'flush')
    at MetricsListener.<anonymous> (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:134:56)
    at step (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:33:23)
    at Object.next (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:14:53)
    at fulfilled (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:5:58)"
}
astuyve commented

Hi @timotheeg - thanks for reaching out!

We print metrics to stdout for customers using the Datadog Lambda Log Forwarder as a data egress mechanism. You can see this logic here. Given that you're using the Datadog Lambda Extension, that shouldn't be enabled unless log forwarding was inadvertently enabled as well with DD_FLUSH_TO_LOG: true.

Can you share the configuration and code for your Lambda function which causes this issue to appear?

Thanks!

timotheeg commented

Thanks @astuyve

We print metrics to stdout for customers using the Datadog Lambda Log Forwarder as a data egress mechanism.

Noted, but so why is there a difference between sendDistributionMetricWithDate() and sendDistributionMetric() on the logging behaviour? When I use sendDistributionMetric(), it does not log, and the data gets sent to DD (no run time error) 🤔 .

that shouldn't be enabled unless log forwarding was inadvertently enabled as well with DD_FLUSH_TO_LOG: true

I can see the lambda DOES have DD_FLUSH_TO_LOG: true in the env vars, I could turn that off and try again 🤔

But also I'll try making a minimal repro case to send you (so there's nothing related to my company in there).