DataDog/datadog-lambda-js

ESModule support failure due to `require`-ing our client code

paco-sparta opened this issue ยท 13 comments

paco-sparta commented

Expected Behavior

Pointing the CMD to handler.handle.mjs should make the lambda compatible with ESModules.

Actual Behavior

When launching our handler with package.json specifying "modules" and pointing our Docker file to "handler.handle.mjs", we see a require error at runtime.

2022-07-20T12:09:01.317Z	undefined	ERROR	Uncaught Exception 	
{
    "errorType": "Error",
    "errorMessage": "require() of ES Module /var/task/app.js from /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js not supported.\nInstead change the require of app.js in /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js to a dynamic import() which is available in all CommonJS modules.",
    "code": "ERR_REQUIRE_ESM",
    "stack": [
        "Error [ERR_REQUIRE_ESM]: require() of ES Module /var/task/app.js from /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js not supported.",
        "Instead change the require of app.js in /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js to a dynamic import() which is available in all CommonJS modules.",
        "    at Module.Hook.Module.require (/var/task/node_modules/dd-trace/packages/dd-trace/src/ritm.js:72:33)",
        "    at _tryRequireSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:248:12)",
        "    at _loadUserAppSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:286:16)",
        "    at loadSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:374:19)",
        "    at Object.<anonymous> (/var/task/node_modules/datadog-lambda-js/dist/handler.js:35:27)",
        "    at _tryRequireFile (file:///var/runtime/index.mjs:857:37)",
        "    at _tryRequire (file:///var/runtime/index.mjs:907:25)",
        "    at _loadUserApp (file:///var/runtime/index.mjs:933:22)",
        "    at Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:964:27)",
        "    at start (file:///var/runtime/index.mjs:1124:42)",
        "    at file:///var/runtime/index.mjs:1130:7",
        "    at async Promise.all (index 0)"
    ]
}

Where app.js is our client code, transpiled to ES2020 using tsc with esmoduleinterop and isolatedModules enabled. Node is configured to use experimental node resolution for imports so file extensions aren't needed.

Steps to Reproduce the Problem

  1. Create a new Docker-based nodejs project with datadog. Use TSC to compile with configurations above.
  2. Point Docker to CMD ["node_modules/datadog-lambda-js/dist/handler.handler.mjs"]
  3. Deploy with serverless
  4. See error above

Specifications

  • Datadog NPM version: 6.81.X
  • Node version: 16.X

Stacktrace

2022-07-20T12:09:01.317Z	undefined	ERROR	Uncaught Exception 	
{
  "errorType": "Error",
  "errorMessage": "require() of ES Module /var/task/app.js from /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js not supported.\nInstead change the require of app.js in /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js to a dynamic import() which is available in all CommonJS modules.",
  "code": "ERR_REQUIRE_ESM",
  "stack": [
      "Error [ERR_REQUIRE_ESM]: require() of ES Module /var/task/app.js from /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js not supported.",
      "Instead change the require of app.js in /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js to a dynamic import() which is available in all CommonJS modules.",
      "    at Module.Hook.Module.require (/var/task/node_modules/dd-trace/packages/dd-trace/src/ritm.js:72:33)",
      "    at _tryRequireSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:248:12)",
      "    at _loadUserAppSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:286:16)",
      "    at loadSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:374:19)",
      "    at Object.<anonymous> (/var/task/node_modules/datadog-lambda-js/dist/handler.js:35:27)",
      "    at _tryRequireFile (file:///var/runtime/index.mjs:857:37)",
      "    at _tryRequire (file:///var/runtime/index.mjs:907:25)",
      "    at _loadUserApp (file:///var/runtime/index.mjs:933:22)",
      "    at Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:964:27)",
      "    at start (file:///var/runtime/index.mjs:1124:42)",
      "    at file:///var/runtime/index.mjs:1130:7",
      "    at async Promise.all (index 0)"
  ]
}
paco-sparta commented

After digging a bit more, it seems like it's not finding package.json, and fails this branch:

datadog-lambda-js/src/runtime/user-function.ts

Lines 130 to 137 in 826c36c

// If package.json type != module, .js files are loaded via require.
const pjHasModule = _hasPackageJsonTypeModule(lambdaStylePath);
if (!pjHasModule) {
const loaded = _tryRequireFile(lambdaStylePath, ".js");
if (loaded) {
return loaded;
}
}

DarcyRaynerDD commented

Hi Paco, is there a package.json next to your app.js file, that specifies it's a ES module?
We try to replicate the logic AWS describes here with how we load modules, and a .js file will be interpreted as a common js file unless there is a package.json file next to it declaring the module type: https://aws.amazon.com/blogs/compute/using-node-js-es-modules-and-top-level-await-in-aws-lambda/

  • ๐Ÿ‘1
paco-sparta commented

I have added a handful of logs to a modified node_modulesto follow the logic:

function _hasFolderPackageJsonTypeModule(folder) {
   ...
    var pj = path_1.default.join(folder, "/package.json");
    var pppp = fs_1.default.existsSync(pj);
    console.log(JSON.stringify({ path: pj, exists: pppp }));

Yields {"path":"/var/task/package.json","exists":true}, and a few lines later in

var pkg = JSON.parse(fs_1.default.readFileSync(pj, "utf-8"));
console.log(JSON.stringify({ pkg, hasModule: pkg.type === "module" }));

I see {"pkg":{...}, "hasModule":true}.

This means that

function _tryRequireSync(appRoot, moduleRoot, module) {
   ...

    // If package.json type != module, .js files are loaded via require.
    var pjHasModule = _hasPackageJsonTypeModule(lambdaStylePath);
    console.log(JSON.stringify({pjHasModule}));
    if (!pjHasModule) {

Returns {"pjHasModule":true} and the logic falls through to require at the end of the method _tryRequireSync, which causes the exception due to incompatibility:

{
    "errorType": "Error",
    "errorMessage": "require() of ES Module /var/task/app.js from /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js not supported.\nInstead change the require of app.js in /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js to a dynamic import() which is available in all CommonJS modules.",
    "trace": [
        "Error [ERR_REQUIRE_ESM]: require() of ES Module /var/task/app.js from /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js not supported.",
        "Instead change the require of app.js in /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js to a dynamic import() which is available in all CommonJS modules.",
        "    at Module.Hook.Module.require (/var/task/node_modules/dd-trace/packages/dd-trace/src/ritm.js:72:33)",
        "    at _tryRequireSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:252:12)",
        "    at _loadUserAppSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:290:16)",
        "    at loadSync (/var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js:378:19)",
        "    at Object.<anonymous> (/var/task/node_modules/datadog-lambda-js/dist/handler.js:35:27)",
        "    at _tryRequireFile (file:///var/runtime/index.mjs:857:37)",
        "    at _tryRequire (file:///var/runtime/index.mjs:907:25)",
        "    at _loadUserApp (file:///var/runtime/index.mjs:933:22)",
        "    at Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:964:27)",
        "    at start (file:///var/runtime/index.mjs:1124:42)",
        "    at file:///var/runtime/index.mjs:1130:7",
        "    at async Promise.all (index 0)"
    ]
}
paco-sparta commented

Let me follow why _tryRequire falls through this path.

EDIT: The lambda environment is picking up the js file with require-style imports loadSync instead of the mjs with load. This is even if I specify the mjs file in the command. WTF.

paco-sparta commented

I have copied the contents of .mjs into .js and another WTF

{
    "errorType": "Runtime.UserCodeSyntaxError",
    "errorMessage": "SyntaxError: Cannot use import statement outside a module",
    "trace": [
        "Runtime.UserCodeSyntaxError: SyntaxError: Cannot use import statement outside a module",
        "    at _loadUserApp (file:///var/runtime/index.mjs:936:17)",
        "    at async Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:964:21)",
        "    at async start (file:///var/runtime/index.mjs:1124:23)",
        "    at async file:///var/runtime/index.mjs:1130:1"
    ]
}

The lambda runtime is not picking up that we're working with modules. What.

paco-sparta commented

I have moved the whole repo to mjs and mts and still no result. I removed the node experimental resolver and still nothing.

My Dockerfile is lean

FROM public.ecr.aws/lambda/nodejs:16

COPY --from=public.ecr.aws/datadog/lambda-extension:latest /opt/extensions/ /opt/extensions

# Configure by --buildArgs
ENV DD_LAMBDA_HANDLER "app.handler"
ENV DD_SITE "datadoghq.eu"
ENV DD_API_KEY "XXXX"

COPY package.json ./
COPY package-lock.json ./

COPY lib ./ # contains a copy of node_modules

CMD ["node_modules/datadog-lambda-js/dist/handler.handler"]

And the serverless one too

provider:
  name: aws
  architecture: x86_64
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'eu-west-1'}
  apiGateway:
    minimumCompressionSize: 1024
    shouldStartNameWithService: true
  environment:
    AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1'
    NODE_OPTIONS: '--enable-source-maps --stack-trace-limit=1000'
  ecr:
    images:
      appimage:
        path: ./
        platform: linux/amd64

I know it's all there.

[12:32:49] ฦ› ls -1
app.mjs
app.mjs.map
node_modules/
package-lock.json
package.json
tracer.mjs
tracer.mjs.map

[12:32:51] ฦ› bat package.json 
   1   โ”‚ {
   2   โ”‚   "name": "XXX",
   3   โ”‚   "version": "1.0.0",
   4   โ”‚   "description": "Serverless aws-nodejs-typescript template",
   5   โ”‚   "author": "YYY",
   6   โ”‚   "license": "SEE LICENSE IN ../LICENSE",
   7   โ”‚   "main": "app.mjs",
   8   โ”‚   "type": "module",

I'm at my wits' end to force it to load the handler. The exception is already telling me that Lambda is using the mjs runtime file:///var/runtime/index.mjs:1130:1

Why why why

DarcyRaynerDD commented

What happens if you change

CMD ["node_modules/datadog-lambda-js/dist/handler.handler"]

to

CMD ["node_modules/datadog-lambda-js/dist/handler.mjs.handler"]

The reason we have two files, is there is a cjs entry point for the node 12 runtime, (which doesn't support .mjs). If you use one of our lambda layers, we make sure node_modules/datadog-lambda-js/dist/handler.handler uses the right version for your runtime, but in the npm module we don't know which runtime you are using, so we leave both in. If this change does work, I will update our documentation to be cleared for Dockerfile users.

If that doesn't work, I think a productive next step would be to send us a zip with a minimum reproduction.

  • ๐Ÿ‘1
paco-sparta commented

Your suggestion was helpful but didn't work :(

Here's the minimum repro I was able to put together: lambda-failure-example.zip

npm i && sls deploy --aws-profile <prof> && sls invoke -f hello --aws-profile <prof> --log --verbose

  • ๐Ÿ‘€1
bgeese-szdm commented

We have the same or a very similar issue. We client code in a container image with modules and no bundling or processing of the JavaScript at all. package.json specifies the type "module" and we have a simply app.js file that exports a handler. That worked fine until we integrated the datadog handler.

The Dockerfile is straightforward:

FROM public.ecr.aws/lambda/nodejs:14

ENV NODE_ENV=production
ENV DD_LAMBDA_HANDLER="app.handler"
ENV DD_TRACE_ENABLED="true"
ENV DD_FLUSH_TO_LOG="true"

# Install Chrome to get all of the dependencies installed
# ...

COPY .npmrc ${LAMBDA_TASK_ROOT}/
COPY package*.json ${LAMBDA_TASK_ROOT}/
COPY src ${LAMBDA_TASK_ROOT}

RUN npm install --production

RUN rm .npmrc

CMD ["node_modules/datadog-lambda-js/dist/handler.handler"]

When we change the CMD back to CMD ["app.handler"], the lambda works, but of course, without the integration we wanted.

Changing to CMD to CMD ["node_modules/datadog-lambda-js/dist/handler.mjs.handler"], as you suggested, does not help:

    "require() of ES modules is not supported.",
    "require() of /var/task/app.js from /var/task/node_modules/datadog-lambda-js/dist/runtime/user-function.js is an ES module file as it is a .js file whose nearest parent package.json contains \"type\": \"module\" which defines all .js files in that package scope as ES modules.",
    "Instead rename app.js to end in .cjs, change the requiring code to use import(), or remove \"type\": \"module\" from /var/task/package.json.",
    "",
DarcyRaynerDD commented

@paco-sparta Thanks for the reproduction, it helps a lot. We will work on a fix for this soon.

  • ๐ŸŽ‰2
kimi-p commented

@paco-sparta @bgeese-szdm
We found the issue is due to node_modules/datadog-lambda-js/dist/handler.js file exists (for node12 backward compatibility, we cannot just remove it) and the existence of the file will make _tryRequire() try to import commonJS first.

The temporary workaround is to remove the file for now. We will think about the Lambda layer and npm use cases, and come up with a solution for the long term. Thank you both for letting us know the issue!

RUN rm node_modules/datadog-lambda-js/dist/handler.js
CMD ["node_modules/datadog-lambda-js/dist/handler.handler"]

image

  • ๐ŸŽ‰5
kimi-p commented

Closing this one now and we will remove this handler.js file later this year after AWS deprecates their support of Node 12. Thanks again for flagging!

  • ๐ŸŽ‰2
pakoito commented

Thank you both for the help!

EDIT: same guy, personal account :D