m3dev/kannon

[Feature Proposal] maximum child job limit

yokomotod opened this issue · 2 comments

yokomotod commented

Now kannon will create unlimited child jobs if they can start.

I would like to control it like

    Kannon(
        # ...
        max_child_jobs=10,
    ).build(task_root)
yokomotod commented

scenario 1: daily task * several years

class DailyTask(TaskOnBullet):
    # for example, download data from BQ for the day
    asof: date = luigi.DateParameter()


class Parent(TaskOnKart):
    from_date: date = luigi.DateParameter()
    to_date: date = luigi.DateParameter()

    def requries(self):
        return [DailyTask(asof=asof) for asof in asof in pd.date_range(self.from_date, self.to_date, freq='D')]

scenario2: I am interested in using TaskOnBullet for all tasks 😈