502 While Deploying (ModuleNotFound Errors)
Opened this issue ยท 7 comments
Context
I am trying to install a project that uses a package called wordcloud
(python). This package has many dependencies, including numpy
. I am using Zappa + Flask. (Slim Handler)
Running Python 3.8
Zappa Settings:
{
"dev": {
"app_function": "app.app",
"aws_region": "us-east-1",
"profile_name": "default",
"project_name": "myproject",
"runtime": "python3.8",
"s3_bucket": "word-cloud-bucket",
"slim_handler": "true"
}
}
Requirements.txt:
argcomplete==1.12.0
boto3==1.14.28
botocore==1.17.28
certifi==2020.6.20
cfn-flip==1.2.3
chardet==3.0.4
click==7.1.2
cycler==0.10.0
docutils==0.15.2
durationpy==0.5
Flask==1.1.2
future==0.18.2
hjson==3.0.1
idna==2.10
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
kappa==0.6.0
kiwisolver==1.2.0
MarkupSafe==1.1.1
matplotlib==3.3.0
numpy==1.19.1
Pillow==7.2.0
pip-tools==5.2.1
placebo==0.9.0
pyparsing==2.4.7
python-dateutil==2.6.1
python-slugify==4.0.1
PyYAML==5.3.1
requests==2.24.0
s3transfer==0.3.3
six==1.15.0
text-unidecode==1.3
toml==0.10.1
tqdm==4.48.0
troposphere==2.6.2
urllib3==1.25.10
Werkzeug==1.0.1
wordcloud==1.7.0
wsgi-request-logger==0.4.6
zappa==0.51.0
https://files.pythonhosted.org/packages/ed/1d/c6d942bc569df4ff1574633b159a68c75f79061fe279975d90f9d2180204/wordcloud-1.7.0-cp37-cp37m-manylinux1_x86_64.whl ; sys_platform == "linux"
https://files.pythonhosted.org/packages/3b/84/7d9ce78ddecd970490f29010b23fbfde0adf35d1568b6d1c3fada9dbb7b5/numpy-1.19.1-pp36-pypy36_pp73-manylinux2010_x86_64.whl ; sys_platform == "linux"
I put in the requests for .whl files at the bottom because I thought that may help. But I don't even know if they are triggering a download.
When running zappa deploy dev
, everthing goes well up to this point:
Deploying API Gateway..
Scheduling..
Unscheduled wordcloudflask-dev-zappa-keep-warm-handler.keep_warm_callback.
Scheduled wordcloudflask-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.
Expected Behavior
Should upload to s3 + Lambda, and use the dependencies correctly.
Actual Behavior
I get the error above, and I can't help but to feel like it has something to do with the way that Zappa is adding/zipping the dependencies. Does it matter that I ran pip install wordcloud
from a venv on my Mac? Would that affect the way things are compiled?
Running zappa tail
, I get this:
[1595825062669] [ERROR] ModuleNotFoundError: No module named 'wordcloud.query_integral_image'
Traceback (most recent call last):
File "/var/task/handler.py", line 609, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 240, in lambda_handler
handler = cls()
File "/var/task/handler.py", line 134, in __init__
self.app_module = importlib.import_module(self.settings.APP_MODULE)
File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/tmp/wordcloudflask/app.py", line 5, in <module>
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
File "/tmp/wordcloudflask/wordcloud/__init__.py", line 1, in <module>
from .wordcloud import (WordCloud, STOPWORDS, random_color_func,
File "/tmp/wordcloudflask/wordcloud/wordcloud.py", line 30, in <module>
from .query_integral_image import query_integral_image
Possible Fix
- The author of wordCloud, when confronted with the missing module error above, often suggests using
conda
. I think there is plenty of conversation around that already, but this is another vote for it. #108 - Instead of seeing what we need based on the
requirements.txt
, why not allow us to package our own dependencies, hand them to you, and have them uploaded to S3?
By conda support, I mean me being able to specify that i need the dependencies installed with conda. I also hope that they are being compiled in the same type of environment as what is on AWS Python Lambdas (some strange form of Linux from what I hear)
Steps to Reproduce
- Create Virtual env
python3 -m venv myEnv
- Activate env
pip install wordcloud
pip freeze > requirements.txt
- Added some .whl files to requirements.txt (first time i didn't do this. Same result)
zappa deploy dev
(slim handler)
Your Environment
- Zappa version used: 0.51.0
- Operating System and Python version: MacOS Catalina, Python 3.8
- The output of
pip freeze
: Above - Link to your project (optional):
- Your
zappa_settings.json
: Above
Try switching (in zappa_setting of course: "runtime": "python3.6") to python 3.6 and see if its working. (maybe 3.7 will also work)
@akobig That actually helped! Now i get to the second problem I have been seeing, which is No module named 'numpy.core._multiarray_umath'
, which I'm guessing must be due to the version of NumPy that I'm using (1.19.1) I will try to set up a new env in 3.6 only
yeah its probably due to a mismatch of your local env python version which is I guess 3.8 and the lambda version which is now 3.6. try installing numpy in a 3.6 local python env, zappa update and tell us if worked I'm interested.
EDIT: It's looking like my issue is related to a known issue with slim_handler when deploying from Windows. I opened a new issue at #2145
I'm running into a similar issue/behavior but not sure if it's related.
I'm encountering an issue returning the error errorMessage": "Unable to import module 'index'"
when using slim_handler : true
. My handler
function is a regular python function that is in index.py
and was otherwise working fine until the package grew to require slim_handler:true
upon installing pandas==1.1.0. I'd expect to find this file in the handler_<project>-<env>-<time>.zip
file but it is missing.
Is there a way to force it to put my index.py file in that zip?
zappa_settings.json:
{
"dev": {
"aws_region": "us-east-1",
"profile_name": "default",
"project_name": "[name]",
"runtime": "python3.6",
"s3_bucket": "[bucket]",
"use_precompiled_packages": true,
"lambda_handler": "index.handler",
"memory_size": 2048,
"timeout_seconds": 300,
"cloudwatch_log_level": "INFO",
"slim_handler": true,
"delete_local_zip": false, // Delete the local zip archive after code updates. Default true.
"delete_s3_zip": false // Delete the s3 zip archive. Default true.
}
}
EDIT: Upon including the file in the zip and uploading manually I find that it is also giving me an import error for pandas which is installed in my venv so in fact I may be having the same issue.
The new error is: Unable to import module 'index': No module named 'pandas'
The index.py that contains my handler is essentially:
import os
import subprocess
import shutil
import time
import json
import pandas as pd
def handler(event, context):
df = pd.DataFrame(columns=['date', 'number', 'url'])
body = df.to_json()
lambda_response = {
"isBase64Encoded": False,
"statusCode": 200,
"headers": {},
"body": json.dumps(body)
}
return lambda_response
@akobig Thank you! That worked! I am very confused why i get 3.8 when i run python --version
from my env (made with pyenv + virtualenv + pyenv-virtualenv), but zappa_settings
is reflecting 3.6 and the deployment worked! I've trying for a week, so thank you very much ๐
Hey @akobig Thanks!
I just changed my python version from 3.8 to 3.6 in zappa_settings.json
and it worked!