Provision to pass queue name (to execute in specified airflow worker) while constructing DatabricksWorkflowTaskGroup
Opened this issue · 0 comments
ckodeboina commented
Use case: Execute all the databricks tasks in dedicated airflow worker.
Problem: The tasks created by "_CreateDatabricksWorkflowOperator" is executing in default airflow worker as is no provision to pass queue value to "_CreateDatabricksWorkflowOperator".
Proposed solution:
Give provision to pass operator parameters or just operator queue while constructing DatabricksWorkflowTaskGroup. So, while internally constructing _CreateDatabricksWorkflowOperator, those parameters can be passed.
Please let me know if any information is required.