Cloudwatch error after setting up our lambda function with CDK construct

[olivierpichon]

Expected Behavior

We use the constructs to instrument our lambda functions. Most of our logs look like this:

START RequestId: ba4d56aa-cb23-468a-960b-5df98d09e9df Version: $LATEST
2022-01-05T10:27:51.431Z	ba4d56aa-cb23-468a-960b-5df98d09e9df	INFO	[dd.trace_id=7576945396481994720 dd.span_id=7576945396481994720] DATADOG TRACER CONFIGURATION - {
    "date": "2022-01-05T10:27:51.430Z",
    "os_name": "Linux",
    "os_version": "4.14.252-207.481.amzn2.x86_64",
    "architecture": "x64",
    "version": "1.1.2",
    "lang": "nodejs",
    "lang_version": "14.18.1",
    "enabled": true,
    "service": "...",
    "agent_url": "http://127.0.0.1:8126",
    "debug": false,
    "sample_rate": 1,
    "sampling_rules": [],
    "tags": {
        "_dd.origin": "lambda",
        "service": "..."
    },
    "log_injection_enabled": true,
    "runtime_metrics_enabled": false,
    "integrations_loaded": [
        "dns",
        "net",
        "pg",
        "http",
        "https",
        "http2"
    ]
}
2022-01-05T10:27:52.491Z	ba4d56aa-cb23-468a-960b-5df98d09e9df	INFO	[dd.trace_id=2744288801532801835 dd.span_id=2744288801532801835] Success
END RequestId: ba4d56aa-cb23-468a-960b-5df98d09e9df

And data is pushed correctly to datadog. We would expect no errors showing up.

Actual Behavior

Quite regularly we get errors in the cloudwatch logs of our newly instrumented lambda function that look like this:

LOGS	Name: datadog-agent	State: Subscribed	Types: [platform,function,extension]
2022-01-05T10:26:42.652Z	undefined	ERROR	{
    "status": "error",
    "message": "datadog:api key not configured, see https://dtdg.co/sls-node-metrics"
}
EXTENSION	Name: datadog-agent	State: Ready	Events: [INVOKE,SHUTDOWN]

We do provide API KEY, via apiKeySecretArn (see below). So not quite sure what does this error means.

Steps to Reproduce the Problem

datadog = new Datadog(scope, id, {
      addLayers: true,
      nodeLayerVersion: 66,
      extensionLayerVersion: 15,
      site: "datadoghq.com",
      enableDatadogTracing: true,
      apiKeySecretArn: ...
    });

datadog.addLambdaFunctions([<LAMBDA_FUNCTIONS>]);

Specifications

• Datadog Lambda Layer version: 15
• Node version: 14

Thx for your help!

[olivierpichon]

At first glance it looks like it could come from https://github.com/DataDog/datadog-lambda-js/blob/master/src/metrics/listener.ts#L172
where the apiKeySecretArn option is not taken into account properly. But I am lacking enough knowledge about how the DD libraries are built to be sure about it.

[tianchu]

@olivierpichon Thanks for your report! Does this error affects any functionality? when using the extension, datadog-lambda-js doesn't actually need the api key at all, only the extension does. So it's a surprise that code path even executes in your case. Do you see the secret arn being set on your lambda as an env var (that's all this library does/cares)? If yes, do you mind moving this issue to datadog-lambda-js instead?

[olivierpichon]

Yes it does set the environment variable correctly. Will move the issue 👍

[hghotra]

Closing this issue, being tracked under datadog-lambda-js instead: DataDog/datadog-lambda-js#256