Support for NodeJS lambda profiling

[vladimirnani]

Expected Behavior

See profiling for NodeJS in lambda

Actual Behavior

Profiling tab remains empty

Steps to Reproduce the Problem

1. Setup APM for serverless NodeJS.
2. Enable DD_PROFILING_ENABLED

Specifications

• Datadog Lambda Layer version: 66
• Node version: 14.x

DATADOG TRACER CONFIGURATION - 
{
    "date": "2021-12-01T12:08:03.119Z",
    "os_name": "Linux",
    "os_version": "4.14.246-198.474.amzn2.x86_64",
    "architecture": "x64",
    "version": "1.5.1",
    "lang": "nodejs",
    "lang_version": "14.18.1",
    "enabled": true,
    "service": "NodeAxiosDataDogExample",
    "agent_url": "http://127.0.0.1:8126",
    "debug": false,
    "sample_rate": 1,
    "sampling_rules": [],
    "tags": {
        "_dd.origin": "lambda",
        "service": "NodeAxiosDataDogExample",
        "runtime-id": "5c661f49-9c7d-435d-a3ea-ecd1954bf3ce"
    },
    "log_injection_enabled": false,
    "runtime_metrics_enabled": false,
    "profiling_enabled": true,
    "integrations_loaded": [
        "http",
        "https"
    ]
}

[agocs]

Thanks for opening this issue, @vladimirnani . Running the profiler in a Lambda environment isn't the same as running it in a serverful environment, so we have some work to do before we can support profiling in Lambda. If I'm not mistaken it's on our roadmap for 2022. I'll leave this issue open so we can track it and update you when profiling is available.

[vvo]

FWIW I was able to do some profiling using https://github.com/serkan-ozal/middy-profiler/ in AWS lambda. Very manual process but it lets do get CPU profiles, store them in S3 and open them later in Chrome. See here: https://twitter.com/vvoyer/status/1498436054851981320

Maybe we could get something similar in Datadog? Not continuous profiling but on-demand profiling?

[astuyve]

Hi, Profiling is now supported in Beta for nodejs by setting the environment variable DD_PROFILING_ENABLED to true. Please ensure you're using the latest version of this library (Layer version 87).

Thanks!

[brainsiq]

@astuyve is anything else needed to enable this? We are not getting profiles from a Node.js Lambda using layer version 87. The functions run for about 20-30s and we noted the comments about 65s in the associated PR so tried setting DD_PROFILING_UPLOAD_PERIOD=10 but still nothing. DataDog UI is showing traces for these functions.

[astuyve]

Followed up in OSS Slack - Pprof files are comparatively heavy (versus metrics, for example), so the profiler buffers data over multiple serial invocations. You should let the profiler run for several lambda-minutes before seeing profile data in Datadog.

Thanks!

[jsteinberg-rbi]

@astuyve So practically speaking the profiler only works with functions that run for several minutes?

[astuyve]

Hi @jsteinberg-rbi - not exactly. The profiler aggregates invocation data across serial lambda invocations, and then flushes data once roughly 60s of execution time has been profiled.

For example - if your function typically runs for ~1s, then we'd expect ~60 invocations to occur before a profile is written back to Datadog. One function invocation doesn't need to run for several minutes.

[steelbrain]

Adding some information here for future stumblers from Google. Here's what I needed to get the profiler working for me

• Set DD_PROFILING_ENABLED to true in lambda env variables
• Add the DD runtime layer to your lambda (arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>:94)
• Add the DD extension layer to your lambda (arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:45)
  • I had not done this and my profiles were never uploaded. I figured out this was the case through help in Slack from DD Employee by setting DD_TRACE_DEBUG=true and DD_LOG_LEVEL=debug

For up to date versions and info, see https://docs.datadoghq.com/serverless/installation/nodejs/?tab=custom