shouldn't log-cleanup task run in sequence?
woorulez opened this issue · 2 comments
Hi, thanks for this great project!
I've just installed log-cleanup
and set airflow_log_cleanup__enable_delete_child_log
to True
.
After a DAG run I can see two log_cleanup_worker's have run successfully.
But the task log tells me that one of the task was skipped. On this particular case the logs/scheduler/
.
[2020-08-04 12:57:47,017] {bash_operator.py:126} INFO - Another task is already deleting logs on this worker node. Skipping it!
[2020-08-04 12:57:47,017] {bash_operator.py:126} INFO - If you believe you're receiving this message in error, kindly check if /tmp/airflow_log_cleanup_worker.lock exists and delete it.
[2020-08-04 12:57:47,017] {bash_operator.py:130} INFO - Command exited with return code 0
I'm new to Airflow, but I think log_cleanup_worker was intended to run in sequence.
Am I seeing this wrong?
I guess I was wrong.
Since my airflow is configured to place scheduler.child_process_log_directory
under core.base_log_folder
,
It's actually wrong to set airflow_log_cleanup__enable_delete_child_log=True
.
I should have stayed with the default setting.
Thanks again for open sourcing a great project!
Glad you liked our project! Feel free to reach out if you have any other questions!