datahub-project/datahub

New release of 'sqlparse' make 'metadata-ingestion' getting error

phongnt1809 opened this issue · 5 comments

phongnt1809 commented

Describe the bug
With version 0.4.4 of sqlparse, sqlparse.keywords.SQL_REGEX does not contain compiled regexes anymore.

Related pull request

Traceback (most recent call last):
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 119, in _add_init_error_context
    yield
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 214, in __init__
    source_class = source_registry.get(source_type)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/api/registry.py", line 173, in get
    tp = self._ensure_not_lazy(key)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/api/registry.py", line 117, in _ensure_not_lazy
    plugin_class = import_path(path)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/api/registry.py", line 48, in import_path
    item = importlib.import_module(module_name)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 57, in <module>
    from datahub.ingestion.source.bigquery_v2.lineage import (
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/source/bigquery_v2/lineage.py", line 39, in <module>
    from datahub.utilities.bigquery_sql_parser import BigQuerySQLParser
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/utilities/bigquery_sql_parser.py", line 6, in <module>
    from datahub.utilities.sql_parser import SqlLineageSQLParser, SQLParser
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/utilities/sql_parser.py", line 9, in <module>
    from datahub.utilities.sql_lineage_parser_impl import SqlLineageSQLParserImpl
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/utilities/sql_lineage_parser_impl.py", line 8, in <module>
    from sqllineage.core.holders import Column, SQLLineageHolder
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/sqllineage/__init__.py", line 41, in <module>
    _monkey_patch()
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/sqllineage/__init__.py", line 35, in _monkey_patch
    _patch_updating_lateral_view_lexeme()
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/sqllineage/__init__.py", line 24, in _patch_updating_lateral_view_lexeme
    if regex("LATERAL VIEW EXPLODE(col)"):
TypeError: 'str' object is not callable

To Reproduce
Ingest bigquery source

jjudy60334 commented

I got the same issue on ingesting bigquery. I upgraded datahub from 0.9.6.1 to 0.10.2, then the ingest error happened.

ecneladis commented

Hi, I got the same issue in 0.9.6.1.

richenc commented

I have the same issue on 0.10.1 for looker and lookml ingestion

ollisala commented

We have a separate loader application, which uses acryl-datahub -package. As a workaround you can pin working versions of dependencies to your wrapper application's requirements.txt:

acryl-datahub==X.X.X
...
sqlparse==0.4.3
sqlfluff==2.0.3
  • 👍1
hsheth2 commented

This was fixed in acryl-datahub v0.10.2.1, which was released yesterday!

  • 👍1