astronomer/astro-sdk

output_table control in "aql.transform"

Andrew-Wichmann opened this issue · 0 comments

Please describe the feature you'd like to see
I would really like to be able to configure the output_table for the TransformOperator when using the @transform decorator.

Describe the solution you'd like

PRODUCTION_DATABASE = Table(
    name="IMAGE",
    conn_id=SNOWFLAKE_CONN_ID,
)

@dag(
    schedule="@daily",
    start_date=datetime(2023, 12, 12),
)
def fetch_missing_files():
    @aql.transform(output_table=Table(conn_id="some_other_db"))
    def fetch_image(prod_database):
        return """
        SELECT image
        FROM {{prod_database}}
        WHERE <condition>
        """
    fetch_image(PRODUCTION_DATABASE)

Are there any alternatives to this feature?
🤔 I'm not sure. I mean, I could just not use the decorator and initialize the Operator myself, but that's not as fun.

Additional context
Nope. I love the product though. I'm really excited to start using more Airflow =)

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)