Parsl/parsl

generator-based task submission with backpressure

benclifford opened this issue · 0 comments

@WardLT made a loop to stop his code from submitting too many (100,000,000) tasks in one go to Parsl, because that uses too much memory. His code intermingles application specific app launching with load management.

There's some librification that could happen here: for example, user supplies a generator (for example a generator comprehension, or a for loop using yield) which provides a long series of tasks; some Parsl library function pulls tasks off that generator to keep Parsl loaded, but not too loaded (for example, by default 10x number of tasks in queues as workers are available).

@WardLT code for reference: https://github.com/HydrogenStorage/molecular-stability-computer/blob/main/compute_emin.py#L147