dbt-labs/dbt-core

[Regression] `dbt --quiet ls --output json` should not print warnings in 1.6 and 1.7

darist opened this issue · 1 comments

darist commented

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I observe this problem with dbt-core 1.7.14. I do not observe this problem with dbt-core 1.7.11

The following command prints a warning, which makes the output invalid json

dbt --quiet ls --output json

[01:19:50  [WARNING]: Deprecated functionality,,User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.,{"name": ...

Expected Behavior

Output should be valid json

dbt --quiet ls --output json

{"name": ...

Steps To Reproduce

profiles.yml file:

foo_profile:
  target: bigquery
  outputs:
    bigquery:
      type: bigquery
      method: "{{ var('method', 'oauth') }}"
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS', '') }}"
      project: "{{ var('data_project', '<redacted>') }}"
      dataset: "{{ var('dataset', 'dbt_' ~ env_var('USER', 'unknown_user')) }}"
      retries: 2
      threads: 8
      timeout_seconds: 300
      location: US

config:
  send_anonymous_usage_stats: False

Then execute

dbt --quiet ls --output json

With dbt-core 1.7.14:

$ dbt --version
Core:
  - installed: 1.7.14
  - latest:    1.7.14 - Up to date!

Plugins:
  - bigquery: 1.7.7  - Update available!
  - postgres: 1.7.14 - Up to date!

$ dbt --quiet ls --output json
01:30:30  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
{"name": "...<redacted>

With dbt-core 1.7.11

$ dbt --quiet ls --output json
{"name": "...<redacted>

Relevant log output

see above

Environment

- OS: Linux
- Python: 3.11.8
- dbt:


$ dbt --version
Core:
  - installed: 1.7.14
  - latest:    1.7.14 - Up to date!

Plugins:
  - bigquery: 1.7.7  - Update available!
  - postgres: 1.7.14 - Up to date!

Which database adapter are you using with dbt?

bigquery

Additional Context

No response

dbeatty10 commented

Thanks for reporting this @darist !

Digging into the details

Looks like the change the introduced the ProjectFlagsMovedDeprecation warning for you was introduced in dbt-core 1.7.14 - May 02, 2024:

  • Move flags from UserConfig in profiles.yml to flags in dbt_project.yml #9183 / #9998

This would also likely be an issue in 1.6 and 1.8 because this is included in these PRs as well:

Things I tried

Attempt 1

Using dbt-core 1.7.14, I tried both of the following, neither of which had any effect:

dbt --warn-error-options '{silence: all}' list -q 
dbt --warn-error-options '{"silence": "ProjectFlagsMovedDeprecation"}' list -q

Log output:

15:16:00  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.

Attempt 2

Attempting the same commands in dbt-core 1.8.0rc1 raised a ValidationError, and the stack trace ended with:

    raise ValidationError(f"{item} is not a valid dbt error name.")
dbt_common.dataclass_schema.ValidationError: P is not a valid dbt error name.

Attempt 3

I also tried adding this to dbt_project.yml:

flags:
  warn_error_options:
    silence:
      - ProjectFlagsMovedDeprecation

This gave a different error:

Runtime Error
  Do not specify both 'config' in profiles.yml and 'flags' in dbt_project.yml. Using 'config' in profiles.yml is deprecated.

Next steps

In the ideal world for dbt-core 1.6 and 1.7, both errors and warnings should be completely silenced by --quiet without users needing to do anything else. But practically, we might have some other considerations that come into play.

Labeling this as refinement for us to figure out how users should silence warnings like this in versions 1.6, 1.7, and 1.8+.