duckdb/dbt-duckdb

Plugin SourceConfig returning column metadata

iamrobo opened this issue · 2 comments

iamrobo commented

I'm trying to make use of the column property in the Source.Table resource type within a custom plugin. Is there a way to access these from a plugin, or will I need to extend the SourceConfig class?

The use case is that I have zipped csv's that do not have column names, and I'd like to load them using the plugin with their associated column names and types.

Below is an example of the source.yml file.

version: 2
sources:
  - name: source_system
    schema: main
    config:
      plugin: zip
      save_mode: ignore
    tables:
      - name: allowance
        identifier: allowance_{{ var('date')}}
        columns:
          - name: Allow_Company
            data_type: text
          - name: Allow_Pointer
            data_type: text
jwills commented

ack, gotcha-- no I don't make that metadata available to the SourceConfig class right now, we will need to add it in; sorry about that!

iamrobo commented
  • 🚀1