Document how to update spark jobs that are called from Airflow DAGs
Opened this issue · 1 comments
adwk67 commented
When upgrading the airflow demo from 23.7 to 23.11 the spark job wrapped by an existing DAG is not updated if the config map containing its definition is changed. This should be investigated and documented as to how to do this.
fhennig commented
I'm assuming this is about this job: https://docs.stackable.tech/home/stable/demos/airflow-scheduled-job
The caching issue might also not be related to just the demo, but can happen to regular users too.