ImportModuleError

Question

ImportModuleError

sarynik opened this issue 3 years ago · 30 comments

I have a rather simple serverless config, with no other plugins, no package exclude patters, yet I get a ModuleImportError upon deployment. The only potential deviation from the most basic example in the README is that I've configured Serverless to build and deploy an image rather than a zip. Any ideas?

{
  "errorType": "Runtime.ImportModuleError",
  "errorMessage": "Error: Cannot find module 's_warmUpPluginWarmer'\nRequire stack:\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js",
  "trace": [
    "Runtime.ImportModuleError: Error: Cannot find module 's_warmUpPluginWarmer'",
    "Require stack:",
    "- /var/runtime/UserFunction.js",
    "- /var/runtime/index.js",
    "    at _loadUserApp (/var/runtime/UserFunction.js:202:13)",
    "    at Object.module.exports.load (/var/runtime/UserFunction.js:242:17)",
    "    at Object.<anonymous> (/var/runtime/index.js:43:30)",
    "    at Module._compile (internal/modules/cjs/loader.js:1085:14)",
    "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)",
    "    at Module.load (internal/modules/cjs/loader.js:950:32)",
    "    at Function.Module._load (internal/modules/cjs/loader.js:790:12)",
    "    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)",
    "    at internal/main/run_main_module.js:17:47"
  ]
}

My Serverless config:

provider:
  name: aws
  region: eu-west-1
  memorySize: 1024
  timeout: 900 #sec
  deploymentBucket:
    name: ${opt:stage}-smp-serverless-deployment
  iam:
    role: ${ssm:/${opt:stage}-lambda-serverless-role-arn}
  ecr:
    images:
      appimage:
        path: ./

plugins:
  - serverless-plugin-warmup

functions:
  main:
    name: ${opt:stage}-conv
    environment:
      INPUT_S3_BUCKET: ${ssm:/${opt:stage}-file-upload-bucket-name}
    image:
      name: appimage

custom:
  warmup:
    warmer:
      enabled: true
    package:
      events:
        - schedule: rate(10 minutes)
      concurrency: 10`

Many thanks in advance!

Answer 1 · 2021-12-18T12:23:40.000Z

Can you share with me the serverless version, warmup plugin version and the command that you use to build the image?

Answer 2 · 2021-12-18T18:54:54.000Z

Serverless version: serverless@2.69.1
Plugin version: serverless-plugin-warmup@6.0.0
Command: serverless deploy --stage testing --region eu-west-1 --org conv --app aws --debug

Answer 3 · 2021-12-20T20:09:50.000Z

I haven't been able to reproduce the issue.
Can you create a repo so I can reproduce the issue?

Answer 4 · 2022-01-03T19:56:34.000Z

Hi @juanjoDiaz
I've added you to the repo:
https://github.com/streamline-uk/sendmypost_lambda_unoconv/invitations

Many thanks for your help

Answer 5 · 2022-01-11T22:11:20.000Z

Hi @streamline-uk ,

Can you invite me again?
The invitation has expired before I could accept it 😅

Answer 6 · 2022-01-11T22:14:37.000Z

@juanjoDiaz done!

Answer 7 · 2022-01-12T17:26:14.000Z

I had to comment out some stuff from you serverless.yaml because I don't have access to your org nor your ssm parameters.
Other than that, everything works on my end doing serverless package which should be enough to replicate the issue.

Answer 8 · 2022-01-13T12:04:21.000Z

Hi @juanjoDiaz
In hindsight, I can see that my report was not very clear. The deployment runs fine, it's just that the warmer Lambda fails to run.

Answer 9 · 2022-01-18T06:23:56.000Z

Hi, i'm facing the same issue.

I can deploy it, but when i try to run/test the Warmer's Lambda, it's fails with the same error

 "errorType": "Runtime.ImportModuleError",
  "errorMessage": "Error: Cannot find module 's_warmUpPluginOfficeHoursWarmer'\nRequire stack:\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js",

I hope it's helps:

I'm using python and due to the size limit of 250MB on unzipped lambda i needed to deleted node_modules folder, so:

[on yml root]

package:
  patterns:
    - "!node_modules/**"
    - "!venv/**"

using that i received the error:

 Serverless Error ----------------------------------------
     No file matches include / exclude patterns

to fix it i included:

[on custom.warmup yml]

custom:
  warmup:
    officeHoursWarmer:
    package:
        individually: true
        patterns:
            - "!**"
            - "../**"
	    - ".warpup/**"

(I tried a lot of combinations, but didn't remember all of them, this one was the last one, based on .READ-ME default patterns)

FIX/WORK-AROUND :

As i needed it working today, i get back to v@5.3.0 and removed the "patterns" from custom.warmup.yml, so its working now:

If its helps you to debug it, that's my working .yml (using v@5.3.0 ) :)

org: xxxxxxxxx
app: xxxxxxxxx
service: xxxxxxxxx

frameworkVersion: "2"

provider:
  name: aws
  runtime: python3.9
  lambdaHashingVersion: 20201221
  apiGateway:
    apiKeys:
      - xxxxxxxxx
  ecr:
    images:
      xxxxxxxxx-image:
        path: ./

functions:
  xxxxxxx:
    image:
      name: xxxxxxx-image
    events:
      - http:
          path: /xxxxxxx/xxxxxxx
          method: get
          private: true

package:
  patterns:
    - "!node_modules/**"
    - "!venv/**"

plugins:
  - serverless-plugin-warmup

custom:
  warmup:
    officeHoursWarmer:
      enabled: true
      events:
        - schedule: cron(0/5 8-18 ? * MON-FRI *)

Answer 10 · 2022-01-19T19:24:55.000Z

The issue can't be reproduced on versión 6.2 of the plugin:

The issue was probably caused by an automatically added layer, which is something that was fixed in version 6.1.

Can you give a newer plugin version a go and let me know?

Answer 11 · 2022-01-20T11:40:45.000Z

@juanjoDiaz I gave it a go but I get the same issue on v6.2. The warmup .js files are missing from the lambda function.
However @fcrdossantos solution of reverting to v5.3.0 (without making any other changes to my setup) worked for me.

Answer 12 · 2022-01-28T07:16:31.000Z

HI @juanjoDiaz even the latest version (with strict fix for v3.0.0) the error continue

I don't know if you tested it with Python, but my stack is based on Python, maybe this can be the problem, i think.

You can do the following:

Create a new serverless project
Create a new Python Script

I created this one named "main.py", please, use it too.

import json
import logging
import pickle
import time

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel

def handler(event, context):
  body = {"hi": "hello"}
  response = {"statusCode": 200, "body": json.dumps(body)}
  return response

Using it you will see that you can't just release a normal lambda because it's bigger than the 250mb max that lambda accepts, so you will need to put in on an ECR (lambda will run it inside docker).

Create a requirements.txt file for python (on root folder)

This will install all Python dependencies:

joblib==1.1.0
numpy==1.22.1
pandas==1.3.5
python-dateutil==2.8.2
pytz==2021.3
scikit-learn==1.0.2
scipy==1.7.3
six==1.16.0
sklearn==0.0
threadpoolctl==3.0.0

Create the Dockerfile to put it

Create a file named "Dockerfile" on root with this content:

FROM public.ecr.aws/lambda/python:3.8

# Recomendation
COPY main.py ${LAMBDA_TASK_ROOT}

COPY requirements.txt  .

RUN  pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"

CMD ["main.handler"]

Until here you can do following this tutorial: https://www.serverless.com/blog/container-support-for-lambda
But try for Python, please

Now create this serverless.yml file

I know it doest not use warm-up yet, but try this file, please

service: testproject-warm
frameworkVersion: "2"

provider:
  name: aws
  runtime: python3.8
  ecr:
    images:
      test-image:
        path: ./

functions:
  main:
    image:
      name: test-image
    events:
      - httpApi:
          path: /
          method: get

Deploy it

You will see its working fine:
✔ Service deployed to stack testproject-warm-dev (238s)

Install serverless-plugin-warmup

I'm using the github version because it's the only one (until now) fixed for serverless 3.0:
npm install git+https://github.com/juanjoDiaz/serverless-plugin-warmup.git

Complete the serverless.yml file:

I put only these lines:

custom:
  warmup:
    officeHoursWarmer:
      enabled: true
      events:
        - schedule: cron(0/5 8-17 ? * MON-FRI *)

So, my full yaml file is:

service: testproject-warm
frameworkVersion: "3"

provider:
  name: aws
  runtime: python3.8
  ecr:
    images:
      test-image:
        path: ./

functions:
  main:
    image:
      name: test-image
    events:
      - httpApi:
          path: /
          method: get

plugins:
  - serverless-plugin-warmup

custom:
  warmup:
    officeHoursWarmer:
      enabled: true
      events:
        - schedule: cron(0/5 8-17 ? * MON-FRI *)

Deploy it:

You will face an error:

Error:
No file matches include / exclude patterns

Done, now you can reproduce the error :)

If you don't want to follow step by step, i uploaded this project here: https://wetransfer.com/downloads/de9ac11826a5320699ea11329ed96ccb20220128071456/f5dec63f53a5551a2b2dd0531e3ec77d20220128071505/8159b4

Just download it, extract and do "serverless deploy"

Oh, the file on wetrasnfer expires on 7 day, so it'll expires in 4th Feb.

Thanks again for your work and suppor here
I hope i can help you now

Answer 13 · 2022-02-01T09:00:45.000Z

@juanjoDiaz I am also getting the same issue with Python deployment.

START RequestId: 8524d91d-14a2-4984-b7f4-a493e352065f Version: $LATEST
2022-02-01T06:38:24.472Z	undefined	ERROR	Uncaught Exception 	
{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module 's_warmUpPluginDefault'\nRequire stack:\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module 's_warmUpPluginDefault'",
        "Require stack:",
        "- /var/runtime/UserFunction.js",
        "- /var/runtime/index.js",
        "    at _loadUserApp (/var/runtime/UserFunction.js:202:13)",
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:242:17)",
        "    at Object.<anonymous> (/var/runtime/index.js:43:30)",
        "    at Module._compile (internal/modules/cjs/loader.js:1085:14)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)",
        "    at Module.load (internal/modules/cjs/loader.js:950:32)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:790:12)",
        "    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)",
        "    at internal/main/run_main_module.js:17:47"
    ]
}

I could see warmUpPluginDefault.zip is empty.. I have unzipped main function package and I could see serverless-plugin-warmup inside node_modules folder. Please can you have a look on it urgently as its affecting so many deploymets

Answer 14 · 2022-02-01T17:58:10.000Z

@fcrdossantos thanks for the detailed description.
I downloaded your package but I still can't reproduce the issue.
It just works for me...

I'm truly confused by this one.
I've tried multiple versions of the plugin and framework.
I've tried local and global install of serverless.
And I still can't reproduce 😕

Which OS and node version are you using?
Might this be a windows-specific issue or a node issue?

The logic to specify the patterns can be found at https://github.com/juanjoDiaz/serverless-plugin-warmup/blob/master/src/config.js#L23-L37

Answer 15 · 2022-02-02T00:33:10.000Z

@juanjoDiaz my pipelines are running through github action.

Operating System
  Ubuntu
  20.04.3
  LTS
  python-version: 3.8
  node-version: 12.x

So when the deployment upload the package to s3 the log is like given below.

functions:
Uploading service auth-xref-io.zip file to S3 (16.62 MB)
Uploading service warmUpPluginDefault.zip file to S3 (1.17 kB)

I have unziped the file and it contains index.js but dont have s_warmUpPluginDefault.js.

minu:default minu.ajith$ ls -la
total 8
drwxr-xr-x@ 3 minu.ajith  staff    96  2 Feb 11:21 .
drwxr-xr-x@ 3 minu.ajith  staff    96  1 Feb 18:30 ..
-rw-r--r--@ 1 minu.ajith  staff  2352  1 Jan  1980 index.js
minu:default minu.ajith$ pwd
/Users/minu.ajith/Downloads/.warmup/default
minu:default minu.ajith$

old warmup plugin zip folder looks different from the new one.

Could you please have a look?

Answer 16 · 2022-02-02T01:06:10.000Z

zip folder dont have s_warmUpPluginDefault.js file thats the reason. It contains only .warmup folder

Answer 17 · 2022-02-02T02:25:23.000Z

Hi @juanjoDiaz thanks for your work and support here :)

I'm using:
OS: Windows 11 - Version 21H2
Python (local): 3.9
Python (AWS): 3.9
Node.JS (local): 16.13.2
Serverless: 3.0.0
Serverless Plugin Warmup: 7.0.0

Answer 18 · 2022-02-02T04:19:39.000Z

@juanjoDiaz Its working fine for me after adding the following under custom.warmup.default

      package:
        individually: true
        patterns:
          - '!../**'
          - '!../../**'
          - ./wsgi_handler.py
          - ./serverless_wsgi.py
          - ./serverless_sdk/*
          - ./s_warmUpPluginDefault.js
          - ./s_api.py

Answer 19 · 2022-02-03T19:59:43.000Z

Are you guys using some Python-related plugin or something else?

If you check this plugin code, the warmers are created under a .warmup'/default/index.js no as ./s_warmUpPluginDefault.js so something must be renaming the file and moving it there....

Answer 20 · 2022-02-06T13:29:18.000Z

@juanjoDiaz I get this issue without using any additional plugins.

I've been able to manually resolve it after each deployment by moving index.js from .warmup/warmer/index.js to lambda root and then changing the handler to index.warmUp. The handler normally defaults to s_warmUpPluginWarmer which is missing from the lambda root folder for plugin v>5.3.0.

It seems from the contents of the file that it's related to this

When using the Serverless Dashboard, the framework automatically injects the serverless_sdk module into your lambda package and wraps your lambda to automatically instrument all the monitoring features in the Dashboard.

@fcrdossantos solution of reverting to v5.3.0 (without making any other changes to my setup) ensured that the file s_warmUpPluginWarmer.js was included in the root folder and the setup was working out of the box, without the manual fix I describe above.

Answer 21 · 2022-02-06T13:45:32.000Z

I can confirm that removing --app and --org arguments from SLS deploy prevents the injection of serverless_sdk and the subsequent modifications to the handler. The warmer works fine without any modifications.

Answer 22 · 2022-02-06T22:08:43.000Z

Thanks @streamline-uk!
We are definitely getting to something.
It seems that the serverless_sdk is doing is injection using a different lifecycle than what this plugin uses.

This is tricky. serverless_sdk is not open-sourced so I have no way to know which lifecycle events they are using, different plugins use different lifecycles making integration difficult (I already have a couple of hacks in the codebase to support the webpack plugin and such) and Serverless framework has never done a good job documenting the lifecycle events and educating on how to use then.

So, we need someone from Serverless framework to document the lifecycle events that they use so I can see how to adapt the code of this plugin.

Answer 23 · 2022-02-07T20:40:05.000Z

Hi all,

I just published 7.0.1 which I think might fix the issue.
I don't use Serverless Dashboard so I can't try.

Can you guys give it a go and let me know if it is fixed?

Answer 24 · 2022-02-16T22:21:36.000Z

Closing since I haven't heard back.

Feel free to reopen if needed.

Answer 25 · 2022-03-05T18:35:58.000Z

Hi @juanjoDiaz
Apologies for the delay. I've just tested it.

The issue now seems to be that the index.js files is buried in the .warmup folder while the default handler path expects it to be in the lambda task root. Manually moving index.js into the root solves the issue.

Answer 26 · 2022-07-03T18:29:46.000Z

Looks like this is still a problem as of today ('s_warmUpPluginDefault' issue) with the plugin running:

Node: 16.7.0
Serverless: 3.19.0
Serverless Plugin Warmup: 7.1.1

Are there any updates here or workaround solutions? Thank you!

Answer 27 · 2022-07-03T20:23:56.000Z

@brendenkrochko there is no version 7.1.1 of this plugin, 7.1.0 is the latest 😅
Are you sure those numbers are correct?

The issue now seems to be that the index.js files is buried in the .warmup folder while the default handler path expects it to be in the lambda task root. Manually moving index.js into the root solves the issue.

@streamline-uk , .warmup/<warmer_name>/index.js is the default location of the warmer code, the lambda is configured to be packaged individually and the warmer lambda is configured to look there.
Can you share your warmer config to check that you are not altering those?
Can you share the generated cloudformation config for the warmer to check as well?
I guess that it's possible the Dashboard might be changing configs at some point since there is no consensus around lifecycle hooks and no transparency on the dashboard stuff.

Answer 28 · 2022-07-04T00:39:03.000Z

@juanjoDiaz Thanks for the prompt response. My mistake on the version number, that was a mistype on my end! 7.1.0 is the one I am on.

So I actually just got everything working on my end using what minu2020 had pasted above (adding in the package options to my config) on Feb 1, 2022. I'm still not entirely sure what the issue was, but everything seems to be alright now.

The warmup config that I am now using successfully is:

warmup:
  default:
    prewarm: true
    package:
      individually: true
      patterns:
        - "!../**"
        - "!../../**"
        - ./serverless_sdk/*
        - ./s_warmUpPluginDefault.js

Answer 29 · 2022-11-02T12:06:17.000Z

Well, I was very sceptical it would work with docker package but hooray, it did!:

  warmup:
    default:
      prewarm: true
      enabled: true
      events:
        - schedule: cron(0/5 1-23 ? * MON-SUN *)
      concurrency: 5
      verbose: true
      logRetentionInDays: 14
      package:
        individually: true
        patterns:
          - "!../**"
          - "!../../**"
          - ./serverless_sdk/*
          - ./s_warmUpPluginDefault.js

Answer 30 · 2023-02-08T01:31:01.000Z

Hi @juanjoDiaz, I was facing this same issue but the proposed workaround didn't work for me. However, I found a fix!

In my case, I'm using both serverless-plugin-warmup for warmups and serverless-esbuild for bundling/minifying.

In the plugins section, if I list serverless-esbuild BEFORE serverless-plugin-warmup, it fails to package the serverless_sdk folder and the s_warmupPluginDefault.js file. If I move serverless-esbuild after serverless-plugin-warmup, however, it packages everything properly.

For example. This does not work:

plugins:
  - serverless-esbuild
  - serverless-plugin-warmup

However, this does work:

plugins:
  - serverless-plugin-warmup
  - serverless-esbuild

I assume it has something to do with lifecycle hooks competing between esbuild and warmup, but I don't know for sure.