Scheduler number of runs

Question

Scheduler number of runs

jozsi opened this issue 5 years ago · 5 comments

Hi,

I noticed that the scheduler has number of runs set to 10.

What was the reason for this? To pick up new DAG files? Cause for that, as far as I understood, dag_dir_list_interval does that automatically every 300 seconds by default.

Answer 1 · 2020-01-31T21:53:47.000Z

This flag makes the scheduler process restart after n runs to avoid weird behaviors

Answer 2 · 2020-01-31T21:57:30.000Z

What are those weird behaviors? Do they still happen with the latest version of Airflow?

Answer 3 · 2020-01-31T23:08:54.000Z

@jozsi scheduler hanging and not scheduling new tasks, but I don't know If this still happen with the current version

Answer 4 · 2020-02-02T12:08:36.000Z

Ref: https://issues.apache.org/jira/plugins/servlet/mobile#issue/AIRFLOW-401

Answer 5 · 2020-02-05T02:15:06.000Z

Ouch, I see a recent comment reporting that the bug still exists in v1.10.4 (dated august 2019; latest is v1.10.7), 3+ years since it has been reported. Hence the workaround is still valid, the only minor drawback could be that it fills up the logs.