sync `airflow.users` does not work in airflow 2.9.0+
Closed this issue ยท 7 comments
Checks
- I have checked for existing issues.
- This report is about the
User-Community Airflow Helm Chart
.
Chart Version
8.8.0
Kubernetes Version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8
Helm Version
version.BuildInfo{Version:"v3.13.3", GitCommit:"c8b948945e52abba22ff885446a1486cb5fd3474", GitTreeState:"clean", GoVersion:"go1.21.5"}
Description
With Airflow 2.9.0 Users
and Roles
have moved once more, so that sync users fails again with:
Table 'ab_permission_view_role' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
The problems seems to be here:
charts/charts/airflow/templates/sync/_helpers/sync_users.tpl
Lines 29 to 35 in 58d96ec
The new location of those classes seems to be this:
https://github.com/apache/airflow/blob/2.9.0/airflow/providers/fab/auth_manager/models/__init__.py
Related issues from last fix:
#747
Relevant Logs
Traceback (most recent call last):
File "/mnt/scripts/sync_users.py", line 147, in <module>
from airflow.auth.managers.fab.models import User, Role
ModuleNotFoundError: No module named 'airflow.auth.managers.fab.models'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/scripts/sync_users.py", line 151, in <module>
from airflow.www.fab_security.sqla.models import User, Role
ModuleNotFoundError: No module named 'airflow.www.fab_security.sqla'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/scripts/sync_users.py", line 154, in <module>
from flask_appbuilder.security.sqla.models import User, Role
File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/sqla/models.py", line 48, in <module>
assoc_permissionview_role = Table(
^^^^^^
File "<string>", line 2, in __new__
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/schema.py", line 596, in __new__
raise exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Custom Helm Values
No response
I've fixed it by changing the template as
# Airflow moves the User and Role models around in different versions
try:
# since 2.9.0
from airflow.providers.fab.auth_manager.models import User, Role
except ModuleNotFoundError:
try:
# from 2.7.0 to 2.8.4
from airflow.auth.managers.fab.models import User, Role
except ModuleNotFoundError:
try:
# from 2.3.0 to 2.6.3
from airflow.www.fab_security.sqla.models import User, Role
except ModuleNotFoundError:
# before 2.3.0
from flask_appbuilder.security.sqla.models import User, Role
Getting uglier with each new version ๐
You have to wonder if they are doing it intentionally to break this chart, lol!
Just in case, since it's for the 2.9.0, I had to update also the pods.tpl
file since the new dockers are using virtual envs so adding the extra pip packaged with the --user
doesn't work.
{{/*
Define an init-container which installs a list of pip packages
*/}}
{{- define "airflow.init_container.install_pip_packages" }}
- name: install-pip-packages
{{- include "airflow.image" . | indent 2 }}
envFrom:
{{- include "airflow.envFrom" . | indent 4 }}
env:
{{- include "airflow.env" . | indent 4 }}
command:
{{- include "airflow.command" . | indent 4 }}
args:
- "bash"
- "-c"
- |
unset PYTHONUSERBASE && \
pip freeze | grep -i {{ range .Values.airflow.protectedPipPackages }}-e {{ printf "%s==" . | quote }} {{ end }} > protected-packages.txt && \
pip install --constraint ./protected-packages.txt {{ range .extraPipPackages }}{{ . | quote }} {{ end }} && \
echo "copying '/home/airflow/.local/' to '/opt/home-airflow-local/.local/'..." && \
rsync -ah --stats --delete /home/airflow/.local/ /opt/home-airflow-local/.local/
volumeMounts:
- name: home-airflow-local
mountPath: /opt/home-airflow-local
{{- end }}
@thesuperzapper - Any updates on this, when this will be merged
@thesuperzapper I deliberately did not add the pod.tmpl modification to this PR (#847 (comment)) as this issue is mainly to sync Users.
Or should we extend the scope?
I will create a separate PR to make extraPipPackages
work, but I have made a permanent fix to the User
/ Role
errors in:
lol
I have made a PR to fix the extraPipPackages
in Airflow 2.9.0: