Question: Is there a way to package node_modules under nodejs folder?

Question

Question: Is there a way to package node_modules under nodejs folder?

cldsnchz opened this issue 4 years ago · 12 comments

Hi, I'm using this plugin in a monorepo project handled with yarn packages.
One of the packages is a layer for external modules. That package just have a package.json with the external dependencies and the serverless.yml template.

Currently the layer is packaged with node_modules in the root folder, so I have to change the NODE_PATH of the services adding /opt/node_modules.

I would like to package the layer with node_modules under a nodejs folder, so the zip is generated with /nodejs/node_modules. In this way I don't have to change the NODE_PATH.

Is there a way to do that, maybe with a hook?

Answer 1 · 2020-06-18T16:21:38.000Z

Here's a working example of a kitchen-sink demo Serverless app that packages with serverless-jetpack that has a layer include nodejs so that consuming code can just do require("normal-package-name") within that:

Layer location: https://github.com/FormidableLabs/aws-lambda-serverless-reference/tree/master/layers/vendor/nodejs (the one you want is the "vendor layer" in the naming for all of these examples)

Serverless config for the layer: https://github.com/FormidableLabs/aws-lambda-serverless-reference/blob/master/serverless.yml#L165-L173

Serverless config for the function that uses the layer: https://github.com/FormidableLabs/aws-lambda-serverless-reference/blob/master/serverless.yml#L141-L160

Code reference in function to import from the layer "like a normal package": https://github.com/FormidableLabs/aws-lambda-serverless-reference/blob/master/src/server/layers.js#L22-L29

Oh, and if you're using tracing mode (which we do recommend) note that it only applies to functions and not layers packaging which use the basic "inspect package.json for prod deps" scheme).

Hope that helps!

Answer 2 · 2020-06-18T19:17:34.000Z

Thanks @ryan-roemer, I took a look to that example but it doesn't seem to fit that well in my project structure (at least I couldn't make it work yet).
I have:

node_modules (the dependencies of all packages are here)
package.json (main project definition with the packages)
packages
|-- external-modules-layer
|   |-- package.json (define the yarn package and have the dependencies to include in the layer)
|   \-- serverless.yml (it has the layer definition)
\-- business-service
    |-- package.json (define the yarn package and have a dev dependency to external-modules-layer)
    \-- serverless.yml (it has the functions definitions pointing to the external-modules-layer)

With this structure everything works fine (including tsc building, sls offline, excluding things from node_modules, etc) but the only issue is that in the layer the node_modules goes to root.

In the example you point me it seems that it is needed the package.json inside the nodejs folder, and the serverless template outside that folder, and also the extra step of creating beforehand the node_modules folder in nodejs

Those requirements doesn't work that well in my project structure, that's why I wanted to know if there is a way to just move the node_modules folder when the zip is created, so I just move it from root to nodejs

If there is no way I guess I will keep adding /opt/node_modules in the NODE_PATH

Thanks!

Answer 3 · 2020-06-18T19:39:27.000Z

Unfortunately, to expand to /opt/nodejs with serverless, you literally need to package a directory somewhere that has the nodejs name and then nodejs/{package.json,node_modules}

One way to get that nodejs directory and package the level above that could be as follows:

packages
\-- business-service
    |-- package.json (define the yarn package and have a dev dependency to external-modules-layer)
    \-- serverless.yml (it has the functions definitions pointing to the external-modules-layer)
layers
\-- external-modules-layer
    |-- nodejs
    |   \-- package.json (define the yarn package and have the dependencies to include in the layer)
    \-- serverless.yml (it has the layer definition)

Then set yarn workspaces something like this:

{
  "workspaces": [
    "packages/*", 
    "layers/*/*"   // You want to install into `nodejs` dir so double star
  ]
}

With a serverless entry for the layer of:

layers:
  {LAYER_NAME}:
    path: .
    name: {NAME}
    jetpack:
      roots:
        - nodejs

The main points are:

The layer.path has to point to one directory up from nodejs dir.
nodejs dir has to be pointed to in yarn workspaces to do the automagic install on root install correctly.
If you're using jetpack you need to point layers.LAYER_NAME.jetpack.roots to the relative path to nodejs to correctly introspect it and include it.

With that you'll get a bundle that has everything prefixed with ./nodejs/FILE_STUFF that should do the correct thing when expanded.

Answer 4 · 2020-06-18T21:05:46.000Z

Thanks @ryan-roemer
I actually tried that (configure the package in nodejs and the serverless.yml one folder up) but didn't work for me, not sure why but node_modules still goes to root.
The configuration I have is:

custom:
  jetpack:
    base: "../.."

layers:
  externalModules:
    path: .
    name: ${self:provider.stage}-${self:service.name}
    jetpack:
      roots:
        - nodejs

Anyway thanks for the help, the question I had is already answered.

If this is something that should work and is not working for me, most probably I'm doing something wrong. If this is the case I will close the ticket

Answer 5 · 2020-06-18T21:17:53.000Z

What are the full paths to nodejs/package.json and serverless.yml?

And what is some sample output from the serverless produced layer zip bundle? (E.g., zipinfo -1 PATH/TO/.serverless/LAYER_BUNDLE_NAME.zip) -- feel free to sanitize names and truncate the list. I just want to get an idea about the shape of things.

Answer 6 · 2020-06-18T22:18:03.000Z

Ok, I attached a simple project to test this jetpack-layer.zip. It just have package that generates the layer.

The structure is:

.
├── layers
│  └── external-modules
│     ├── nodejs
│     │  └── package.json
│     └── serverless.yml
├── lerna.json
├── package.json
└── yarn.lock

The jetpack related part of the serverless.yml

custom:
  jetpack:
    base: ../..

layers:
  externalModules:
    path: .
    name: ${self:provider.stage}-${self:service.name}
    jetpack:
      roots:
        - nodejs

the result:

$ unzip -l layers/external-modules/.serverless/externalModules.zip | head
Archive:  layers/external-modules/.serverless/externalModules.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     1579  01-01-1980 00:00   node_modules/.bin/uuid
       94  01-01-1980 00:00   node_modules/@middy/core/.babelrc
      670  01-01-1980 00:00   node_modules/@middy/core/CHANGELOG.md
     1152  01-01-1980 00:00   node_modules/@middy/core/LICENSE
     2147  01-01-1980 00:00   node_modules/@middy/core/README.md
    22486  01-01-1980 00:00   node_modules/@middy/core/__tests__/index.js
      631  01-01-1980 00:00   node_modules/@middy/core/__tests__/isPromise.js

Answer 7 · 2020-06-18T22:27:34.000Z

Oh snap, I think I know what's up. Is the contents of layers/external-modules/nodejs/node_modules empty (I think you alluded that this is the case)?

If so, then here's what's happening. When jetpack has base: below function/layer root, those lower-than-root includes get smashed into the root of the zip file because a zip archive has no concept of "below root". So there's really nothing in nodejs that is used and it's smashed down there. Here's a link to a full discussion of the situation jetpack docs: https://github.com/FormidableLabs/serverless-jetpack#packaging-files-outside-cwd (also collapsed.bail is a great option to guarantee nothing conflicts when using a monorepo).

Let me think of potential solutions to this. (Looking back at our reference project we have this that I forgot about: https://github.com/FormidableLabs/aws-lambda-serverless-reference/blob/master/package.json#L9

"postinstall": "cd layers/vendor/nodejs && yarn",

that presumably takes care of this situation by not doing yarn/lerna hoisting and forcing everything to be in the full path (there it's layers/vendor/nodejs/node_modules)

Answer 8 · 2020-06-18T22:44:51.000Z

That's correct, layers/external-modules/nodejs/node_modules is empty, that's why I added base: ../.., to use the root's node_modules

Answer 9 · 2020-06-18T23:04:52.000Z

OK, I think I have a solution -- using nohoist to keep the modules in the correct location while still also being part of the yarn workspaces:

Diff:

diff --git a/layers/external-modules/nodejs/package.json b/layers/external-modules/nodejs/package.json
index 8c811c1..c43ea18 100644
--- a/layers/external-modules/nodejs/package.json
+++ b/layers/external-modules/nodejs/package.json
@@ -2,6 +2,9 @@
   "name": "@layers/external-modules",
   "version": "1.0.0",
   "private": true,
+  "workspaces": {
+    "nohoist": ["**"]
+  },
   "license": "ISC",
   "scripts": {
     "clean": "rimraf ../.serverless",
diff --git a/layers/external-modules/serverless.yml b/layers/external-modules/serverless.yml
index 6fc0863..27df8c0 100644
--- a/layers/external-modules/serverless.yml
+++ b/layers/external-modules/serverless.yml
@@ -17,6 +17,14 @@ layers:
   externalModules:
     path: .
     name: ${self:provider.stage}-${self:service.name}
+    package:
+      exclude:
+        # Serverless adds a built-in exclude for all layers at `layers.{NAME}.path`
+        # for multi-part builds. This just undoes that so exclude doesn't apply and
+        # we can package normally.
+        #
+        # **Note**: Not needed normally unless `path: .`
+        - "!./**"
     jetpack:
       roots:
         - nodejs`

Sample output (notice the --report dumps the internal zip paths. Previously the nodejs/node_modules things were ../../node_modules which were then collapsed);

$ node ../../node_modules/.bin/serverless jetpack package --report
## ... stuff snipped variously ...

- nodejs/node_modules/@middy/core/package.json
- nodejs/node_modules/aws-sdk/package.json
- nodejs/node_modules/base64-js/package.json
- nodejs/node_modules/buffer/package.json
- nodejs/node_modules/depd/package.json
- nodejs/node_modules/events/package.json
- nodejs/node_modules/fast-safe-stringify/package.json
- nodejs/node_modules/http-errors/package.json
- nodejs/node_modules/ieee754/package.json
- nodejs/node_modules/inherits/package.json
- nodejs/node_modules/isarray/package.json
- nodejs/node_modules/jmespath/package.json
- nodejs/node_modules/lambda-log/package.json
- nodejs/node_modules/once/package.json
- nodejs/node_modules/punycode/package.json
- nodejs/node_modules/querystring/.package.json.un~
- nodejs/node_modules/querystring/package.json
- nodejs/node_modules/sax/package.json
- nodejs/node_modules/setprototypeof/package.json
- nodejs/node_modules/statuses/package.json
- nodejs/node_modules/toidentifier/package.json
- nodejs/node_modules/url/package.json
- nodejs/node_modules/uuid/package.json
- nodejs/node_modules/wrappy/package.json
- nodejs/node_modules/xml2js/node_modules/sax/package.json
- nodejs/node_modules/xml2js/package.json
- nodejs/node_modules/xmlbuilder/package.json
- nodejs/package.json

and we can verify in the zip output:

$ zipinfo -1 .serverless/externalModules.zip | grep package.json
nodejs/node_modules/@middy/core/package.json
nodejs/node_modules/aws-sdk/package.json
nodejs/node_modules/base64-js/package.json
nodejs/node_modules/buffer/package.json
nodejs/node_modules/depd/package.json
nodejs/node_modules/events/package.json
nodejs/node_modules/fast-safe-stringify/package.json
nodejs/node_modules/http-errors/package.json
nodejs/node_modules/ieee754/package.json
nodejs/node_modules/inherits/package.json
nodejs/node_modules/isarray/package.json
nodejs/node_modules/jmespath/package.json
nodejs/node_modules/lambda-log/package.json
nodejs/node_modules/once/package.json
nodejs/node_modules/punycode/package.json
nodejs/node_modules/querystring/.package.json.un~
nodejs/node_modules/querystring/package.json
nodejs/node_modules/sax/package.json
nodejs/node_modules/setprototypeof/package.json
nodejs/node_modules/statuses/package.json
nodejs/node_modules/toidentifier/package.json
nodejs/node_modules/url/package.json
nodejs/node_modules/uuid/package.json
nodejs/node_modules/wrappy/package.json
nodejs/node_modules/xml2js/node_modules/sax/package.json
nodejs/node_modules/xml2js/package.json
nodejs/node_modules/xmlbuilder/package.json
nodejs/package.json

Answer 10 · 2020-06-18T23:32:48.000Z

Thanks for taking a look.

Using nohoist is what I wanted to avoid, actually I started using jetpack because the default serverless package doesn't work with hoist packages.

Anyway is good to see at least a way to have this thing working, I tried with nohoist but didn't notice that I had to add exclude: "!./**" to make it work.

Answer 11 · 2020-06-19T00:27:07.000Z

Yeah, serverless doesn't work at all out of the box with monorepos and such.

The nohoist as my diff have it is just limited to only external-modules/nodejs/package.json and I wouldn't imagine that you'd have other packages symlinked in there from within the monorepo so my thought was that everything else should be fine. Is there some use case where just the external modules layer package needs to not be nohoist-ed? (I'm just curious generally for community support as not a lot of folks use layers -- I've just got them for demo / testing).

Answer 12 · 2020-06-19T12:53:05.000Z

An example of an internal package in a layer can be when you don't want to put shared code in the layer itself to for example avoid using import ... from /opt/shared from other packages because that doesn't work that well with tsc or local tests, etc.

A way to fix that problem is to have a package shared with the code, import it as import ... from shared in the services, and then add it as dependency in a layer package if you want to deploy it in a layer instead of in each service.

In any case a hoist layer package can be easily implemented packaging node_modules in root. The only thing is needed is to add /opt/node_modules in the NODE_PATH of the services which use that layer. That is what I wanted to avoid but is not a big deal.

Anyway, you are right, it seems to me that layers are not used that much.