when slim_handler is used MySQLdb cannot be found by Django
Closed this issue · 24 comments
this issue is similar to #755
when using the zappa_settings.json configuration to turn on "use_precompiled_packages":true and then turning on "slim_handler": true I get error below:
1491418915817] Error loading MySQLdb module: libmysqlclient.so.18: cannot open shared object file: No such file or directory: ImproperlyConfigured Traceback (most recent call last): File "/var/task/handler.py", line 508, in lambda_handler return LambdaHandler.lambda_handler(event, context) File "/var/task/handler.py", line 239, in lambda_handler handler = cls() File "/var/task/handler.py", line 142, in __init__ wsgi_app_function = get_django_wsgi(self.settings.DJANGO_SETTINGS) File "/var/task/django_zappa_app.py", line 20, in get_django_wsgi return get_wsgi_application() File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/core/wsgi.py", line 14, in get_wsgi_application File "/tmp/dailyfantasyser/django/__init__.py", line 18, in setup apps.populate(settings.INSTALLED_APPS) File "/tmp/dailyfantasyser/django/apps/registry.py", line 108, in populate app_config.import_models(all_models) File "/tmp/dailyfantasyser/django/apps/config.py", line 198, in import_models self.models_module = import_module(models_module_name) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/contrib/auth/models.py", line 41, in <module> File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/models/base.py", line 139, in __new__ File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/models/base.py", line 324, in add_to_class File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/models/options.py", line 250, in contribute_to_class File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/__init__.py", line 36, in __getattr__ File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/utils.py", line 240, in __getitem__ File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/utils.py", line 111, in load_backend File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/backends/mysql/base.py", line 27, in <module> ImproperlyConfigured: Error loading MySQLdb module: libmysqlclient.so.18: cannot open shared object file: No such file or directory
I am running in virtualenv on macos10.12.3 with python2.7, Django 1.8.5.
Expected Behavior
the error should not occur and the library should be found in the proper location for Django to run properly.
Actual Behavior
the following error: ImproperlyConfigured: Error loading MySQLdb module: libmysqlclient.so.18: cannot open shared object file: No such file or directory
Possible Fix
Steps to Reproduce
- update project to use_precompiled_packages:true and slim_handler:true
- deploy large project using zappa update staging
- run zappa tail to view the log
Your Environment
- Zappa version used: 0.40.0p
- Operating System and Python version: macos10.12.3
- Link to your project (optional):
- Your
zappa_settings.py:
{ "staging": { "aws_region": "us-east-1", "django_settings": "daily.settings_lambda", "s3_bucket": "daily", "memory_size": 1024, "timeout_seconds": 120, "use_precompiled_packages": true, "slim_handler": true, "delete_local_zip": false, "log_level": "DEBUG", } }
Does this change need any documentation updates? Will Django users' know they need to set some value here in order for their project to work under certain conditions?
It should JustWork^tm, at least for MySQL
I think this is fixed in 0.42.0
https://github.com/Miserlou/Zappa/blob/master/zappa/handler.py#L110
Why libmysqlclient.so.18 is in default.
I have a project that does not depend upon MySQL client but Postgres. And, it was failing at the runtime with a very less obvious error of Failed to find library...right filename?.
@gagandeep have you tried to use the "include" feature in your zappa_settings file to include the proper Postgres library? It seems like it might be preferable to have zappa check your Django settings and automatically include the proper library based on the database. Maybe this is a future enhancement.
@timj98 Yes, I had to change include. My only point is Library of any kind shouldn't assume things whether its MySQL or Postgres doesn't matter.
If someone needs to include mysql with Zappa it should be mentioned in include setting. Zappa shouldn't add libmysqlclient.so.18 as others might be using something else and that will break things.
Django & MySQL with slim handler should be part of the documentation as an example, not the library default.
All good points. I think that the reason it was included by default was so that it just worked when someone was using Django & MySQL. It sounds like it should be reworked to check the Django settings and include the default library from there so one doesn't have to manually include anything.
@gagandeep Did you have to specify a specific postgresql library? Did you just have an empty include=[] in your zappa_settings.json?
Yes, I have empty include in my zappa settings. Postgres is working fine with that.
how is it going this issue? I have a same issue on zappa==0.51.0, django==3.0.5, python==3.7
Same here.
I am now having this problem as well. I have an existing Django 3/Zappa 0.51/Postgres 10 project that works fine. I created a new project, copying the old project's zappa_settings.json and tweaking values as appropriate, but it can't deploy at all. I'm now concerned that this problem will crop up on my old project at some point, which is an issue since it's a production critical environment. I have never had an include or exclude in my zappa_settings.json and adding one to the new project doesn't seem to help.
Yes, you got it absolutely right. That's why I wrote earlier
Why libmysqlclient.so.18 is in default.
i still have issues tried all these solutions and no luck
I also still have issues. When I add include:[] it does not work and the code still tries to import libmysqlclient.so.18
adding include : [] does not help. How can I add postgres to include?
"include": [] in the Zappa_settings.json file didn't work for me either.
I'd guess the
"include": []hack works due to
Line 105 in ba20c85
Looks like the "include": [] hack doesn't work for some (including me) due to this:
Lines 2378 to 2380 in 818f7de
since the
INCLUDE= var is not added regardless of whether you omit "include" or you write "include": [] in zappa_settings.json.
I propose using a sentinal value "include": null to signify that you don't want to include the default, error-causing value ['libmysqlclient.so.18']
Something like this:
include = self.stage_config.get('include', [])
if include is None:
settings_s += 'INCLUDE=[]\n'
elif len(include) >= 1:
settings_s += "INCLUDE=" + str(include) + '\n'Thoughts?
I remember trying “include”: null. It didn’t work for me either
This doesn't work for me either. It's trying to include a library I don't need which is hardcoded into line 105 of handler.py.
Line 105 in ba20c85
I'm using postgres and as far as I'm aware my project doesn't use this libmysqlclient, yet its causing my scripts to fail when it cannot find it,. Is there a way to remove it since it's not needed?