datacoves/dbt-coves

[BUG] Add `sqlalchemy` as a dependency

Closed this issue · 5 comments

mjkanji commented

Describe the bug
#449 imports sqlalchemy here:

https://github.com/datacoves/dbt-coves/blame/05934394f470e621fa2d08a9ca7ff536eb834b20/dbt_coves/tasks/data_sync/sql_database/__init__.py#L10

But it's not added as a dependency in pyproject.toml. This is leading to a ModuleNotFoundError: No module named 'sqlalchemy' error.

BAntonellini commented

Hey @mjkanji , could you provide us the steps to reproduce such ModuleNotFoundError using dbt-coves?

jakedave commented

I'm having the same issue; our requirements.txt is simply:

dbt_coves

As of 3 weeks ago, we started seeing ModuleNotFoundError: No module named 'sqlalchemy'; seemingly, dbt_coves used to import this for you

specifically, we are using your fivetran extractor:

dbt-coves extract fivetran --api-key <API_KEY> --api-secret <API_SECRET> --path <PATH>

This seems to be something introduced from 1.7.3 -> 1.7.4; in our case, this is such a minor usecase, we might just pin down for the time being

jakedave commented

Update: Pinning down does not work either. Using 1.7.2 or 1.7.3 comes up with a new issue:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.19/x64/bin/dbt-coves", line 5, in <module>
    from dbt_coves.core.main import main
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/dbt_coves/core/main.py", line 21, in <module>
    from dbt_coves.tasks.setup.main import SetupTask
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/dbt_coves/tasks/setup/main.py", line 7, in <module>
    from .pre_commit import SetupPrecommitTask
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/dbt_coves/tasks/setup/pre_commit.py", line 3, in <module>
    import copier
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/copier/__init__.py", line 6, in <module>
    from .main import *  # noqa: F401,F403
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/copier/main.py", line 28, in <module>
    from .subproject import Subproject
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/copier/subproject.py", line [15](https://github.com/ArcadiaPower/fivetran/actions/runs/8743474427/job/23994085442#step:5:16), in <module>
    from .template import Template
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/copier/template.py", line 18, in <module>
    from yamlinclude import YamlIncludeConstructor
ModuleNotFoundError: No module named 'yamlinclude'Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.19/x64/bin/dbt-coves", line 5, in <module>
    from dbt_coves.core.main import main
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/dbt_coves/core/main.py", line 21, in <module>
    from dbt_coves.tasks.setup.main import SetupTask
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/dbt_coves/tasks/setup/main.py", line 7, in <module>
    from .pre_commit import SetupPrecommitTask
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/dbt_coves/tasks/setup/pre_commit.py", line 3, in <module>
    import copier
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/copier/__init__.py", line 6, in <module>
    from .main import *  # noqa: F401,F403
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/copier/main.py", line 28, in <module>
    from .subproject import Subproject
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/copier/subproject.py", line [15](https://github.com/ArcadiaPower/fivetran/actions/runs/8743474427/job/23994085442#step:5:16), in <module>
    from .template import Template
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/copier/template.py", line 18, in <module>
    from yamlinclude import YamlIncludeConstructor
ModuleNotFoundError: No module named 'yamlinclude'
BAntonellini commented

@jakedave Copier had a critical issue some weeks ago by "blindly" upgrading pyyaml-include: copier-org/copier#1568

It was fixed in 1.7.4 by pinning pyyaml-include down, and if you're having issues in 1.7.4 because you don't have SQLAlchemy it'll be solved in #456

It should be released today. For the time being, if you urgently need to run dbt-coves==1.7.3 please add pyyaml-include==1.3.2 as a dependency

jakedave commented

Thanks @BAntonellini ! I am using the latest version (1.7.7 at the time of writing) and this has resolved the issue. Appreciate the fix and explanation!

  • ❤️1