apache/superset

Helm upgrade from 4.0.2 -> 4.1.1 fails as psycopg2 is no longer bundled into lean Superset image

kirmorozov1992 opened this issue · 12 comments

kirmorozov1992 commented

Bug description

Hi
I want to upgrade Superset 4.0.2 to 4.1.1 version using Helm.
helm upgrade --install superset superset/superset -f values.yaml
I got issue with psycopg2. Help, please.

Screenshots/recordings

Defaulted container "superset-init-db" out of: superset-init-db, wait-for-postgres (init)
Upgrading DB schema...
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
2024-11-22 03:34:11,278:ERROR:superset.app:Failed to create app
Traceback (most recent call last):
File "/app/superset/app.py", line 40, in create_app
app_initializer.init_app()
File "/app/superset/initialization/init.py", line 476, in init_app
self.setup_db()
File "/app/superset/initialization/init.py", line 667, in setup_db
pessimistic_connection_handling(db.engine)
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 998, in engine
return self.get_engine()
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 1017, in get_engine
return connector.get_engine()
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 594, in get_engine
self._engine = rv = self._sa.create_engine(sa_url, options)
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 1027, in create_engine
return sqlalchemy.create_engine(sa_url, **engine_opts)
File "", line 2, in create_engine
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 544, in create_engine
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 811, in dbapi
import psycopg2
ModuleNotFoundError: No module named 'psycopg2'
Traceback (most recent call last):
File "/usr/local/bin/superset", line 8, in
sys.exit(superset())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1685, in invoke
super().invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 355, in decorator
app = __ctx.ensure_object(ScriptInfo).load_app()
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 309, in load_app
app = locate_app(import_name, name)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 238, in locate_app
return find_app_by_string(module, app_name)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 166, in find_app_by_string
app = attr(*args, **kwargs)
File "/app/superset/app.py", line 40, in create_app
app_initializer.init_app()
File "/app/superset/initialization/init.py", line 476, in init_app
self.setup_db()
File "/app/superset/initialization/init.py", line 667, in setup_db
pessimistic_connection_handling(db.engine)
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 998, in engine
return self.get_engine()
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 1017, in get_engine
return connector.get_engine()
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 594, in get_engine
self._engine = rv = self._sa.create_engine(sa_url, options)
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 1027, in create_engine
return sqlalchemy.create_engine(sa_url, **engine_opts)
File "", line 2, in create_engine
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 544, in create_engine
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 811, in dbapi
import psycopg2
ModuleNotFoundError: No module named 'psycopg2'

Superset version

master / latest-dev

Python version

3.9

Node version

16

Browser

Chrome

Additional context

No response

Checklist

  • I have searched Superset docs and Slack and didn't find a solution to my problem.
  • I have searched the GitHub issue tracker and didn't find a similar bug report.
  • I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
dosubot commented

Hello @kirmorozov1992! I'm here to assist you with any bugs, questions, or contributions you have. Let's work together to solve the issue you're facing.

To resolve the "No module named 'psycopg2'" error when upgrading Apache Superset using Helm, you need to ensure that the psycopg2 package is installed as part of the bootstrap script in your Helm chart configuration. You can do this by adding the installation command for psycopg2 in the bootstrapScript section of your values.yaml file. Here is an example configuration:

bootstrapScript: |
  #!/bin/bash
  pip install psycopg2==2.9.6 \
    sqlalchemy-bigquery==1.6.1 \
    elasticsearch-dbapi==0.2.5 &&\
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

This script ensures that psycopg2 is installed when the Superset application starts, which should resolve the module not found error [1].

To continue talking to Dosu, mention @dosu.

  • 👍1
  • 👎4
kirmorozov1992 commented

@dosu Unfortunately it does not help me. Somebody has other advice?

dosubot commented

This is a tough nut to crack, could really use your help on this, @rusackas.

To continue talking to Dosu, mention @dosu.

  • 👍1
  • 👎2
DavidProdinger commented

There is the same issue for the docker compose version (docker-compose-image-tag.yml), not limited to Helm.

paulchoi commented

This worked for me. You need psycopg2-binary.
You can ignore clickhouse-connect.

bootstrapScript: |
  #!/bin/bash
  pip install \
    "clickhouse-connect>=0.6.8" \
    "psycopg2-binary>=2.9.10" \
    &&\
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi