[Bug] `dbt clone` not redirecting exception to logs
jp-dbt opened this issue · 1 comments
Is this a new bug in dbt-core?
- I believe this is a new bug in dbt-core
- I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
Any errors that occur while running dbt clone
are written to stderr instead of stdout.
Expected Behavior
Any errors that occur while running dbt clone
are written to stdout, which is the case for other dbt commands.
Steps To Reproduce
- Modify a project so that an error occurs. In my case I added a non-existent doc ref for a column in a model so that a parsing error occurs
- Run
dbt parse 1>out.txt 2>err.txt
. Theout.txt
file contains the whole output including the error whileerr.txt
is empty - Run
dbt clone 1>out.txt 2>err.txt
. Theout.txt
file contains the non-error dbt output and err.txt contains the error and stacktrace
Relevant log output
No response
Environment
- OS: MacOS Sonoma 14.4.1, Debian 10
- Python: 3.11.9
- dbt: 1.7.1
Which database adapter are you using with dbt?
No response
Additional Context
Possibly related: errors during dbt clone
are not written to the dbt.log
file.
Thanks for reporting this @jp-dbt !
Root cause
It appears that the root cause is dbt clone
failing to capture the underlying exception and redirect it to the logs. Rather, the exception is being raised directly to the user via standard error (stderr).
See below for the details of an extremely simple reproducible example.
Reprex
Replace postgres
below with the name of your dbt connection profile. The last line is invalid, and it will raise an error for most dbt commands.
dbt_project.yml
name: "my_project"
profile: "postgres"
give_me: an_error
Run these commands:
dbt parse 1>parse-out.txt 2>parse-err.txt
dbt clone 1>clone-out.txt 2>clone-err.txt
Expected output
Here's what the contents of *-out.txt
should be for nearly every command:
[0m15:36:22 Running with dbt=1.8.0
[0m15:36:22 Encountered an error:
Runtime Error
at path []: Additional properties are not allowed ('give_me' was unexpected)
Error encountered in /path/to/dbt/project/dbt_project.yml
But dbt clone
has a big Python stacktrace in *-err.txt
instead:
Traceback (most recent call last):
File "/Users/dbeatty/projects/environments/dbt_1.8/bin/dbt", line 8, in <module>
sys.exit(cli())
...
File "/Users/dbeatty/projects/environments/dbt_1.8/lib/python3.10/site-packages/dbt/config/project.py", line 390, in create_project
raise ProjectContractError(e) from e
dbt.exceptions.ProjectContractError: Runtime Error
at path []: Additional properties are not allowed ('give_me' was unexpected)
Error encountered in /path/to/dbt/project/dbt_project.yml
Checking other dbt commands
Here's the script that I used to test a large range of dbt commands (pretty much everything but dbt init
, I think:
check_stdout_and_stderr.sh
check_stdout_and_stderr.sh
#!/bin/bash
# Define an array of dbt commands
commands=(
"build"
"clean"
"clone"
"compile"
"debug"
"deps"
"list"
"parse"
"retry"
"run"
"run-operation"
"seed"
"show"
"snapshot"
"test"
"docs generate"
"docs serve"
"source freshness"
"source snapshot-freshness"
)
# Loop through each command in the array
for cmd in "${commands[@]}"; do
echo "Running dbt $cmd..."
# Replace spaces with hyphens for file names
filename=$(echo "$cmd" | tr ' ' '-')
# Execute dbt with the current command, redirect stdout and stderr to files
dbt $cmd 1>${filename}-out.txt 2>${filename}-err.txt
# Use wc to count lines, words, and characters in output and error files
wc ${filename}-out.txt
wc ${filename}-err.txt
done
Run these commands:
chmod +x check_stdout_and_stderr.sh
./check_stdout_and_stderr.sh
Output
Running dbt build...
6 25 267 build-out.txt
0 0 0 build-err.txt
Running dbt clean...
6 25 267 clean-out.txt
0 0 0 clean-err.txt
Running dbt clone...
1 4 41 clone-out.txt
600 1584 47342 clone-err.txt
Running dbt compile...
6 25 267 compile-out.txt
0 0 0 compile-err.txt
Running dbt debug...
43 154 1803 debug-out.txt
0 0 0 debug-err.txt
Running dbt deps...
6 25 267 deps-out.txt
0 0 0 deps-err.txt
Running dbt list...
6 25 267 list-out.txt
0 0 0 list-err.txt
Running dbt parse...
6 25 267 parse-out.txt
0 0 0 parse-err.txt
Running dbt retry...
6 25 267 retry-out.txt
0 0 0 retry-err.txt
Running dbt run...
6 25 267 run-out.txt
0 0 0 run-err.txt
Running dbt run-operation...
0 0 0 run-operation-out.txt
4 15 112 run-operation-err.txt
Running dbt seed...
6 25 267 seed-out.txt
0 0 0 seed-err.txt
Running dbt show...
6 25 267 show-out.txt
0 0 0 show-err.txt
Running dbt snapshot...
6 25 267 snapshot-out.txt
0 0 0 snapshot-err.txt
Running dbt test...
6 25 267 test-out.txt
0 0 0 test-err.txt
Running dbt docs generate...
6 25 267 docs-generate-out.txt
0 0 0 docs-generate-err.txt
Running dbt docs serve...
6 25 267 docs-serve-out.txt
0 0 0 docs-serve-err.txt
Running dbt source freshness...
6 25 267 source-freshness-out.txt
0 0 0 source-freshness-err.txt
Running dbt source snapshot-freshness...
6 25 267 source-snapshot-freshness-out.txt
0 0 0 source-snapshot-freshness-err.txt