louisguitton/dbt-metadata-utils

dbt snapshots are not supported?

hoanghapham opened this issue · 2 comments

Hi Louis,

I'm following your guide in this post to build a search engine for my dbt project, but currently I'm stuck at make update-git-metadata and make update-index. The error messages I have are in this form:

nodes -> snapshot.internal_analytics.snapshot_src_pipedrive_organizations -> resource_type
  value is not a valid enumeration member; permitted: 'model', 'analysis', 'test', 'operation', 'seed', 'source' (type=type_error.enum; enum_values=[<DbtResourceType.model: 'model'>, <DbtResourceType.analysis: 'analysis'>, <DbtResourceType.test: 'test'>, <DbtResourceType.operation: 'operation'>, <DbtResourceType.seed: 'seed'>, <DbtResourceType.source: 'source'>])

Does this mean that snapshots are not supported yet?

HI @hoanghapham ,
sorry I've missed your issue and that I'm replying so late.

quickfix

This looks like simple enough to fix. The issue is that I lacked an example dbt repo with dbt snapshots to test on.
Depending on how the JSON artifact looks for that dbt resource type, it might be very easy to add support.
Here

class DbtResourceType(str, Enum):
"""Different types of dbt resources."""
model = "model"
analysis = "analysis"
test = "test"
operation = "operation"
seed = "seed"
source = "source"
, one would need to add

+    snapshot = "snapshot"

longterm

FWIW, since my post, dbt has added clear jsonschema specifications for their schemas (see https://schemas.getdbt.com/dbt/manifest/v1.json ) so I could rewrite those pydantic classes to match that more...

for you

Feel free to try to add the quickfix locally on your version of the code and try it on your example dbt project. if it works, I welcome PRs ! :)

thanks Louis, I will try this.