Airflow sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object
naani10 opened this issue · 9 comments
Checks
- I have checked for existing issues.
- This report is about the
User-Community Airflow Helm Chart
.
Chart Version
8.7.1
Kubernetes Version
AKS- 1.25.6
Helm Version
version.BuildInfo{Version:"v3.7.2", GitCommit:"663a896f4a815053445eec4153677ddc24a0a361", GitTreeState:"clean", GoVersion:"go1.16.10"}
Description
I am trying to install airflow 1.26.0 using 8.7.1 helm chart
I am getting errors in sync user pod and airflow web pod
airflow-sync-users-b8dc65576-j55b5.log
airflow-web-7469d4449b-h46gd.log
Relevant Logs
airflow sync logs
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:468 FutureWarning: The 'dag_default_view' setting in [webserver] has the old default value of 'tree'. This value has been changed to 'grid' in the running config, but please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:468 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:347 DeprecationWarning: The auth_backend option in [api] has been renamed to auth_backends - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:468 FutureWarning: The 'log_id_template' setting in [elasticsearch] has the old default value of '{dag_id}-{task_id}-{execution_date}-{try_number}'. This value has been changed to '{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}' in the running config, but please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:370 DeprecationWarning: The auth_backend option in [api] has been renamed to auth_backends - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:384 FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:338 DeprecationWarning: The sql_alchemy_pool_enabled option in [core] has been moved to the sql_alchemy_pool_enabled option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:345 DeprecationWarning: The sql_alchemy_pool_size option in [core] has been moved to the sql_alchemy_pool_size option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:357 DeprecationWarning: The sql_alchemy_max_overflow option in [core] has been moved to the sql_alchemy_max_overflow option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:363 DeprecationWarning: The sql_alchemy_pool_recycle option in [core] has been moved to the sql_alchemy_pool_recycle option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:370 DeprecationWarning: The sql_alchemy_pool_pre_ping option in [core] has been moved to the sql_alchemy_pool_pre_ping option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:401 DeprecationWarning: The sql_engine_encoding option in [core] has been moved to the sql_engine_encoding option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/retries.py:28 DeprecationWarning: The max_db_retries option in [core] has been moved to the max_db_retries option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/base.py:27 DeprecationWarning: The sql_alchemy_schema option in [core] has been moved to the sql_alchemy_schema option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:338 DeprecationWarning: The sql_alchemy_pool_enabled option in [core] has been moved to the sql_alchemy_pool_enabled option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:345 DeprecationWarning: The sql_alchemy_pool_size option in [core] has been moved to the sql_alchemy_pool_size option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:357 DeprecationWarning: The sql_alchemy_max_overflow option in [core] has been moved to the sql_alchemy_max_overflow option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:363 DeprecationWarning: The sql_alchemy_pool_recycle option in [core] has been moved to the sql_alchemy_pool_recycle option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:370 DeprecationWarning: The sql_alchemy_pool_pre_ping option in [core] has been moved to the sql_alchemy_pool_pre_ping option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:401 DeprecationWarning: The sql_engine_encoding option in [core] has been moved to the sql_engine_encoding option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py:395 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py:3471 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
Traceback (most recent call last):
File "/mnt/scripts/sync_users.py", line 141, in <module>
flask_app = www_app.create_app()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 129, in create_app
init_api_experimental_auth(flask_app)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/extensions/init_security.py", line 60, in init_api_experimental_auth
auth = import_module(backend.strip())
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/airflow/keycloak_auth.py", line 27, in <module>
from flask_appbuilder.security.sqla.models import User
File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/sqla/models.py", line 48, in <module>
assoc_permissionview_role = Table(
File "<string>", line 2, in __new__
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
return fn(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 596, in __new__
raise exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Airflow web pod logs
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:468 FutureWarning: The 'dag_default_view' setting in [webserver] has the old default value of 'tree'. This value has been changed to 'grid' in the running config, but please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:468 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:347 DeprecationWarning: The auth_backend option in [api] has been renamed to auth_backends - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:468 FutureWarning: The 'log_id_template' setting in [elasticsearch] has the old default value of '{dag_id}-{task_id}-{execution_date}-{try_number}'. This value has been changed to '{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}' in the running config, but please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:370 DeprecationWarning: The auth_backend option in [api] has been renamed to auth_backends - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:384 FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:338 DeprecationWarning: The sql_alchemy_pool_enabled option in [core] has been moved to the sql_alchemy_pool_enabled option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:345 DeprecationWarning: The sql_alchemy_pool_size option in [core] has been moved to the sql_alchemy_pool_size option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:357 DeprecationWarning: The sql_alchemy_max_overflow option in [core] has been moved to the sql_alchemy_max_overflow option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:363 DeprecationWarning: The sql_alchemy_pool_recycle option in [core] has been moved to the sql_alchemy_pool_recycle option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:370 DeprecationWarning: The sql_alchemy_pool_pre_ping option in [core] has been moved to the sql_alchemy_pool_pre_ping option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:401 DeprecationWarning: The sql_engine_encoding option in [core] has been moved to the sql_engine_encoding option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py:905 DeprecationWarning: The namespace option in [kubernetes] has been moved to the namespace option in [kubernetes_executor] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/base.py:27 DeprecationWarning: The sql_alchemy_schema option in [core] has been moved to the sql_alchemy_schema option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/retries.py:28 DeprecationWarning: The max_db_retries option in [core] has been moved to the max_db_retries option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py:396 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py:3483 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py:124 DeprecationWarning: The processor_poll_interval option in [scheduler] has been renamed to scheduler_idle_sleep_time - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:338 DeprecationWarning: The sql_alchemy_pool_enabled option in [core] has been moved to the sql_alchemy_pool_enabled option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:345 DeprecationWarning: The sql_alchemy_pool_size option in [core] has been moved to the sql_alchemy_pool_size option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:357 DeprecationWarning: The sql_alchemy_max_overflow option in [core] has been moved to the sql_alchemy_max_overflow option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:363 DeprecationWarning: The sql_alchemy_pool_recycle option in [core] has been moved to the sql_alchemy_pool_recycle option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:370 DeprecationWarning: The sql_alchemy_pool_pre_ping option in [core] has been moved to the sql_alchemy_pool_pre_ping option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:401 DeprecationWarning: The sql_engine_encoding option in [core] has been moved to the sql_engine_encoding option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/retries.py:28 DeprecationWarning: The max_db_retries option in [core] has been moved to the max_db_retries option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:338 DeprecationWarning: The sql_alchemy_pool_enabled option in [core] has been moved to the sql_alchemy_pool_enabled option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:345 DeprecationWarning: The sql_alchemy_pool_size option in [core] has been moved to the sql_alchemy_pool_size option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:357 DeprecationWarning: The sql_alchemy_max_overflow option in [core] has been moved to the sql_alchemy_max_overflow option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:363 DeprecationWarning: The sql_alchemy_pool_recycle option in [core] has been moved to the sql_alchemy_pool_recycle option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:370 DeprecationWarning: The sql_alchemy_pool_pre_ping option in [core] has been moved to the sql_alchemy_pool_pre_ping option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:401 DeprecationWarning: The sql_engine_encoding option in [core] has been moved to the sql_engine_encoding option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/base.py:27 DeprecationWarning: The sql_alchemy_schema option in [core] has been moved to the sql_alchemy_schema option in [database] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py:396 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py:3483 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
[�[34m2023-05-16 10:58:26,537�[0m] {�[34mkeycloak_auth.py:�[0m37} INFO�[0m - Initializing Custom Api Auth Backend�[0m
/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py:124 DeprecationWarning: The processor_poll_interval option in [scheduler] has been renamed to scheduler_idle_sleep_time - the old setting has been used, but please update your config.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/__main__.py", line 7, in <module>
run()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 58, in __init__
self.setup(app)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 118, in setup
self.app.wsgi()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/util.py", line 412, in import_app
app = app(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 176, in cached_app
app = create_app(config=config, testing=testing)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 153, in create_app
init_appbuilder(flask_app)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/extensions/init_appbuilder.py", line 648, in init_appbuilder
AirflowAppBuilder(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/extensions/init_appbuilder.py", line 160, in __init__
self.init_app(app, session)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/extensions/init_appbuilder.py", line 206, in init_app
self.sm = self.security_manager_class(self)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 193, in __init__
super().__init__(appbuilder)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/fab_security/sqla/manager.py", line 68, in __init__
user_datamodel = SQLAInterface(self.user_model)
File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/models/sqla/interface.py", line 68, in __init__
for prop in class_mapper(obj).iterate_properties:
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/base.py", line 448, in class_mapper
mapper = _inspect_mapped_class(class_, configure=configure)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/base.py", line 427, in _inspect_mapped_class
mapper._check_configure()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/mapper.py", line 1941, in _check_configure
_configure_registries({self.registry}, cascade=True)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/mapper.py", line 3527, in _configure_registries
_do_configure_registries(registries, cascade)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/mapper.py", line 3566, in _do_configure_registries
mapper._post_configure_properties()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/mapper.py", line 1958, in _post_configure_properties
prop.init()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/interfaces.py", line 231, in init
self.do_init()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/relationships.py", line 2150, in do_init
self._process_dependent_arguments()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/relationships.py", line 2245, in _process_dependent_arguments
self.target = self.entity.persist_selectable
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/relationships.py", line 2112, in entity
argument = self._clsregistry_resolve_name(self.argument)()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/clsregistry.py", line 393, in _resolve_name
rval = d[token]
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/_collections.py", line 746, in __missing__
self[key] = val = self.creator(key)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/clsregistry.py", line 352, in _access_cls
return _determine_container(key, decl_class_registry[key])
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/clsregistry.py", line 313, in _determine_container
value = value.attempt_get([], key)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/clsregistry.py", line 144, in attempt_get
raise exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Multiple classes found for path "Permission" in the registry of this declarative base. Please use a fully module-qualified path.
Custom Helm Values
No response
Dear community,
Can any one help me with this issue.
Is there any workaround to avoid this errors?
Thanks
Yeah we are also facing the same issue. Anyone able to find workaround?
Please check your Airflow version as there might be several deprecated variables that need to be updated. You can verify the variables using your Airflow.
It is recommended to review the changes made after version 2.3.0, as there have been significant updates.
You can find more information on the official Apache Airflow documentation: link.
@naani10 @PankushSharma @crotielwf I have a suspicion that the issue is related to the webserver_config.py
.
Are you using a custom webserver_config.py
(which is configured with the web.webserverConfig.*
values)?
If so, can you share it here?
If you are not using a custom webserver_config.py
, I bet that our default one is not working properly.
Can try setting the webserver config to a slightly different default by using the values:
web:
webserverConfig:
stringOverride: |
from airflow.www.fab_security.manager import AUTH_DB
AUTH_TYPE = AUTH_DB
@naani10 actually looking at your logs, the issue is in your custom /opt/airflow/keycloak_auth.py
file.
The issue is the same as this user's problem (apache/airflow#27138), where they were importing from flask_appbuilder.security.sqla.models import User
, but the correct User
module for airflow 2.3.0+ is from airflow.www.fab_security.sqla.models import User
.
Hi @thesuperzapper please find the webserver_config.py below.
import os
from flask_appbuilder.security.manager import AUTH_DB
# from flask_appbuilder.security.manager import AUTH_LDAP
# from flask_appbuilder.security.manager import AUTH_OAUTH
# from flask_appbuilder.security.manager import AUTH_OID
# from flask_appbuilder.security.manager import AUTH_REMOTE_USER
basedir = os.path.abspath(os.path.dirname(__file__))
# Flask-WTF flag for CSRF
WTF_CSRF_ENABLED = True
# ----------------------------------------------------
# AUTHENTICATION CONFIG
# ----------------------------------------------------
# For details on how to set up each of the following authentication, see
# http://flask-appbuilder.readthedocs.io/en/latest/security.html# authentication-methods
# for details.
# The authentication type
# AUTH_OID : Is for OpenID
# AUTH_DB : Is for database
# AUTH_LDAP : Is for LDAP
# AUTH_REMOTE_USER : Is for using REMOTE_USER from web server
# AUTH_OAUTH : Is for OAuth
AUTH_TYPE = AUTH_DB
# Uncomment to setup Full admin role name
AUTH_ROLE_ADMIN = 'Admin'
# Uncomment to setup Public role name, no authentication needed
AUTH_ROLE_PUBLIC = 'Op'
# Will allow user self registration
# AUTH_USER_REGISTRATION = True
# The recaptcha it's automatically enabled for user self registration is active and the keys are necessary
# RECAPTCHA_PRIVATE_KEY = PRIVATE_KEY
# RECAPTCHA_PUBLIC_KEY = PUBLIC_KEY
# Config for Flask-Mail necessary for user self registration
# MAIL_SERVER = 'smtp.gmail.com'
# MAIL_USE_TLS = True
# MAIL_USERNAME = 'yourappemail@gmail.com'
# MAIL_PASSWORD = 'passwordformail'
# MAIL_DEFAULT_SENDER = 'sender@gmail.com'
# The default user self registration role
# AUTH_USER_REGISTRATION_ROLE = "Public"
# When using OAuth Auth, uncomment to setup provider(s) info
# Google OAuth example:
# OAUTH_PROVIDERS = [{
# 'name':'google',
# 'token_key':'access_token',
# 'icon':'fa-google',
# 'remote_app': {
# 'api_base_url':'https://www.googleapis.com/oauth2/v2/',
# 'client_kwargs':{
# 'scope': 'email profile'
# },
# 'access_token_url':'https://accounts.google.com/o/oauth2/token',
# 'authorize_url':'https://accounts.google.com/o/oauth2/auth',
# 'request_token_url': None,
# 'client_id': GOOGLE_KEY,
# 'client_secret': GOOGLE_SECRET_KEY,
# }
# }]
# When using LDAP Auth, setup the ldap server
# AUTH_LDAP_SERVER = "ldap://ldapserver.new"
# When using OpenID Auth, uncomment to setup OpenID providers.
# example for OpenID authentication
# OPENID_PROVIDERS = [
# { 'name': 'Yahoo', 'url': 'https://me.yahoo.com' },
# { 'name': 'AOL', 'url': 'http://openid.aol.com/<username>' },
# { 'name': 'Flickr', 'url': 'http://www.flickr.com/<username>' },
# { 'name': 'MyOpenID', 'url': 'https://www.myopenid.com' }]
# ----------------------------------------------------
# Theme CONFIG
# ----------------------------------------------------
# Flask App Builder comes up with a number of predefined themes
# that you can use for Apache Airflow.
# http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes
# Please make sure to remove "navbar_color" configuration from airflow.cfg
# in order to fully utilize the theme. (or use that property in conjunction with theme)
APP_THEME = "bootstrap-theme.css" # default bootstrap
# APP_THEME = "amelia.css"
# APP_THEME = "cerulean.css"
# APP_THEME = "cosmo.css"
# APP_THEME = "cyborg.css"
# APP_THEME = "darkly.css"
# APP_THEME = "flatly.css"
# APP_THEME = "journal.css"
# APP_THEME = "lumen.css"
# APP_THEME = "paper.css"
# APP_THEME = "readable.css"
# APP_THEME = "sandstone.css"
# APP_THEME = "simplex.css"
# APP_THEME = "slate.css"
# APP_THEME = "solar.css"
# APP_THEME = "spacelab.css"
# APP_THEME = "superhero.css"
# APP_THEME = "united.css"
# APP_THEME = "yeti.css"
@naani10 while you probably should update your webserver_config.py
to use the new import locations for things like AUTH_LDAP
, that is NOT the cause of your pods failing.
The issue is caused by your custom script at /opt/airflow/keycloak_auth.py
. Clearly, it's not compatible with airflow since 2.3.0 for the reasons described in #747 (comment).
You must either remove that script if you don't need it, or update it to support newer airflow.
Hello everyone, I have the same issue but with a custom security manager.
I'm using airflow 2.6.1 and helm chart 8.7.1
The code of webserver_config.py:
import os
from flask_appbuilder.security.manager import AUTH_DB
from my_modules.security import CustomSecurityManager
AUTH_TYPE = AUTH_DB
FAB_SECURITY_MANAGER_CLASS = CustomSecurityManager
and the code of the security manager:
from airflow.www.security import AirflowSecurityManager
class CustomSecurityManager(AirflowSecurityManager):
def auth_user_oauth(self, userinfo):
return super().auth_user_oauth(userinfo)
If I comment FAB_SECURITY_MANAGER_CLASS and the import it works.
Thanks
This was working after we updated below configs
Thanks @thesuperzapper for support
web:
webserverConfig:
stringOverride: |
from airflow.www.fab_security.manager import AUTH_DB
AUTH_TYPE = AUTH_DB