Option to not execute on-run-end statements when running unit tests
tsaastam opened this issue · 1 comments
I have a SQL pipeline I'm converting to use dbt. The pipeline runs in Snowflake and it uses a lot of temporary tables, which are not directly supported by dbt's Snowflake implementation. For this reason I am instead using transient tables (which are relatively cheap in Snowflake, but permanent) and then dropping them at the end of the run. This is done with an on-run-end
directive in my dbt_project.yml
:
on-run-end:
- "drop table if exists {{ ref('my_temp_table_1') }};"
- "drop table if exists {{ ref('my_temp_table_2') }};"
The problem is that when I run my unit tests, dbt also wants to run these drop statements. While I could make this work, it would be much nicer to have an option somewhere to only run the on-run-end
statements when I'm not running a unit test - there's no reason to (try to) drop actual tables inside a unit test. How can I accomplish this?
I've discovered a macro, dbt_unit_testing.running_unit_test()
, which could in theory be used to only run the drop table
statements when the current run is not a unit test:
on-run-end: |
{% if dbt_unit_testing.running_unit_test() %}
select 1 from dual;
{% else %}
drop table if exists {{ ref('my_temp_table_1') }};
drop table if exists {{ ref('my_temp_table_2') }};
{% endif %}
However, the macro doesn't seem to exist at the time when the on-run-end
hook is called:
17:15:12.474796 [error] [MainThread]: Encountered an error:
Compilation Error in operation <my_project>-on-run-end-0 (./dbt_project.yml)
'dict object' has no attribute 'running_unit_test'. This can happen when calling a macro that does not exist. Check for typos and/or install package dependencies with "dbt deps".
Is it possible to configure the dbt-unit-testing package in such a way that I could use this macro from inside dbt_project.yml
? Or otherwise, how can I accomplish what I want to do? Is this something that would require additional functionality in dbt itself?
Thank you,
Ah, I have actually just discovered how to use the standard flags provided by dbt itself to accomplish what I need. In dbt_project.yml
:
on-run-end: |
{% if flags.WHICH == "test" %}
select 1 from dual;
{% else %}
drop table if exists {{ builtins.ref('my_temp_table_1') }};
drop table if exists {{ builtins.ref('my_temp_table_2') }};
{% endif %}
So this issue can be closed.
Perhaps this example could be added to some documentation somewhere, to make it easier for other people to figure out. :)