cal-itp/data-infra

Unpin Python Version from Linting CI Task Once dbt Formally Supports Python 3.12

Closed this issue · 0 comments

User story / feature request

As a contributor, I would like the sqlfluff-lint task in CI to run successfully on the latest available version of Python, something that is not possible for 3.12.1 at the time of writing.

Acceptance Criteria

Once Python 3.12 is fully and officially supported by the dbt-core Python package, we should be able to remove the code introduced in this commit.

Notes

In different places across the Cal-ITP ecosystem of tools and services, different Python versions are in use. Airflow, for instance, uses a version corresponding to its managed Composer image (3.8.2 at the time of writing). Several different CI tasks designate versions (3.10, etc.) that correspond to healthy function using their given imports. Our linting task takes the latest available Python release for the sake of staying up to date.

The linting GitHub action failed when using Python 3.12.1 because 3.12.1 removed distutils, which was referenced by an underlying dbt function called by the sqlfluff-lint task. dbt-core does not formally support Python 3.12 yet, but the changes between 3.11 and 3.12 were relatively minimal for functionality used by dbt-core. This appears to be the only related breakage impacting Cal-ITP projects, and will be resolved once dbt formally supports Python 3.12 (technically >=3.12.1, since 3.12.0 didn't fully implement the breaking change). At that point, we can remove the version pin we put in place as part of #3175.