output_table control in "aql.transform"
Andrew-Wichmann opened this issue · 0 comments
Andrew-Wichmann commented
Please describe the feature you'd like to see
I would really like to be able to configure the output_table
for the TransformOperator when using the @transform
decorator.
Describe the solution you'd like
PRODUCTION_DATABASE = Table(
name="IMAGE",
conn_id=SNOWFLAKE_CONN_ID,
)
@dag(
schedule="@daily",
start_date=datetime(2023, 12, 12),
)
def fetch_missing_files():
@aql.transform(output_table=Table(conn_id="some_other_db"))
def fetch_image(prod_database):
return """
SELECT image
FROM {{prod_database}}
WHERE <condition>
"""
fetch_image(PRODUCTION_DATABASE)
Are there any alternatives to this feature?
🤔 I'm not sure. I mean, I could just not use the decorator and initialize the Operator myself, but that's not as fun.
Additional context
Nope. I love the product though. I'm really excited to start using more Airflow =)
Acceptance Criteria
- All checks and tests in the CI should pass
- Unit tests (90% code coverage or more, once available)
- Integration tests (if the feature relates to a new database or external service)
- Example DAG
- Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
- Exception handling in case of errors
- Logging (are we exposing useful information to the user? e.g. source and destination)
- Improve the documentation (README, Sphinx, and any other relevant)
- How to use Guide for the feature (example)