sync `airflow.users` does not work in airflow 2.9.0+

Checks

I have checked for existing issues.
This report is about the User-Community Airflow Helm Chart.

Chart Version

8.8.0

Kubernetes Version

Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8

Helm Version

version.BuildInfo{Version:"v3.13.3", GitCommit:"c8b948945e52abba22ff885446a1486cb5fd3474", GitTreeState:"clean", GoVersion:"go1.21.5"}

Description

With Airflow 2.9.0 Users and Roles have moved once more, so that sync users fails again with:

Table 'ab_permission_view_role' is already defined for this MetaData instance.  Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

The problems seems to be here:

charts/charts/airflow/templates/sync/_helpers/sync_users.tpl

Lines 29 to 35 in 58d96ec

    
           try: 
        
             # since 2.7.0 
        
             from airflow.auth.managers.fab.models import User, Role 
        
           except ModuleNotFoundError: 
        
             try: 
        
               # from 2.3.0 to 2.6.3 
        
               from airflow.www.fab_security.sqla.models import User, Role

The new location of those classes seems to be this:
https://github.com/apache/airflow/blob/2.9.0/airflow/providers/fab/auth_manager/models/__init__.py

Related issues from last fix:
#747

Relevant Logs

Traceback (most recent call last):
  File "/mnt/scripts/sync_users.py", line 147, in <module>
    from airflow.auth.managers.fab.models import User, Role
ModuleNotFoundError: No module named 'airflow.auth.managers.fab.models'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/scripts/sync_users.py", line 151, in <module>
    from airflow.www.fab_security.sqla.models import User, Role
ModuleNotFoundError: No module named 'airflow.www.fab_security.sqla'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/scripts/sync_users.py", line 154, in <module>
    from flask_appbuilder.security.sqla.models import User, Role
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/sqla/models.py", line 48, in <module>
    assoc_permissionview_role = Table(
                                ^^^^^^
  File "<string>", line 2, in __new__
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/schema.py", line 596, in __new__
    raise exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance.  Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

Custom Helm Values

No response

I've fixed it by changing the template as

# Airflow moves the User and Role models around in different versions
try:
  # since 2.9.0
  from airflow.providers.fab.auth_manager.models import User, Role
except ModuleNotFoundError:
  try:
    # from 2.7.0 to 2.8.4
    from airflow.auth.managers.fab.models import User, Role
  except ModuleNotFoundError:
    try:
      # from 2.3.0 to 2.6.3
      from airflow.www.fab_security.sqla.models import User, Role
    except ModuleNotFoundError:
      # before 2.3.0
      from flask_appbuilder.security.sqla.models import User, Role

Getting uglier with each new version 😅

You have to wonder if they are doing it intentionally to break this chart, lol!

Just in case, since it's for the 2.9.0, I had to update also the pods.tpl file since the new dockers are using virtual envs so adding the extra pip packaged with the --user doesn't work.

{{/*
Define an init-container which installs a list of pip packages
*/}}
{{- define "airflow.init_container.install_pip_packages" }}
- name: install-pip-packages
  {{- include "airflow.image" . | indent 2 }}
  envFrom:
    {{- include "airflow.envFrom" . | indent 4 }}
  env:
    {{- include "airflow.env" . | indent 4 }}
  command:
    {{- include "airflow.command" . | indent 4 }}
  args:
    - "bash"
    - "-c"
    - |
      unset PYTHONUSERBASE && \
      pip freeze | grep -i {{ range .Values.airflow.protectedPipPackages }}-e {{ printf "%s==" . | quote }} {{ end }} > protected-packages.txt && \
      pip install --constraint ./protected-packages.txt {{ range .extraPipPackages }}{{ . | quote }} {{ end }} && \
      echo "copying '/home/airflow/.local/' to '/opt/home-airflow-local/.local/'..." && \
      rsync -ah --stats --delete /home/airflow/.local/ /opt/home-airflow-local/.local/
  volumeMounts:
    - name: home-airflow-local
      mountPath: /opt/home-airflow-local
{{- end }}

@thesuperzapper - Any updates on this, when this will be merged

@thesuperzapper I deliberately did not add the pod.tmpl modification to this PR (#847 (comment)) as this issue is mainly to sync Users.
Or should we extend the scope?

I will create a separate PR to make extraPipPackages work, but I have made a permanent fix to the User / Role errors in:

#857

lol

I have made a PR to fix the extraPipPackages in Airflow 2.9.0:

#858

	try:
	# since 2.7.0
	from airflow.auth.managers.fab.models import User, Role
	except ModuleNotFoundError:
	try:
	# from 2.3.0 to 2.6.3
	from airflow.www.fab_security.sqla.models import User, Role