dbt-labs/dbt-utils

unpivot: add parameters for explicitly defining columns to pivot on, rather than just listing columns to exclude.

JustGitting opened this issue · 0 comments

Describe the feature

I have a table with a few hundred columns and want to unpivot only 10 columns into two columns (key, value) and keep 3 other columns as "index" columns.

With the current unpivot() there doesn't appear to be an easy way to explicitly define the 10 columns that need to unpivoted and to keep the 3 columns.

unpivot() has the following parameters. My understanding is to achieve my goal, I need to pass the other 100-odd columns to the exclude parameter, which is very inconvenient.

{{ dbt_utils.unpivot(
  relation=ref('table_name'),
  cast_to='datatype',
  exclude=[<list of columns to exclude from unpivot>],
  remove=[<list of columns to remove>],
  field_name=<column name for field>,
  value_name=<column name for value>
) }}

It would make unpivot() easier to use if two parameters were added:

"keep" - column names to be kept unchanged.
"include" - column names to be unpivoted.

Describe alternatives you've considered

I looked at using get_filtered_columns_in_relation() to get the list of columns for exclusion, but again I need manually list columns to exclude and there does not appear to do pattern matching. In R it's easy to use starts_with() (https://tidyselect.r-lib.org/reference/starts_with.html) for example to do pattern matching of fields.

Additional context

N/A

Who will this benefit?

All DBT users.

Are you interested in contributing this feature?

No.