How to dynamically add jobs to a running workflow?
Opened this issue · 6 comments
Hello, I am trying to figure out how to program the following workflow:
- step 1: do some work and store the results in a DB table (lots of rows).
- step 2: read the results from the DB table, and process them but in batches of 1000 for example, because the total number of rows to process is way too high for a single job.
The problem is that when defining step 2, I cannot know at definition time what is the data to process, and how many rows there are, therefore how many jobs I need to spawn to process everything in batches.
I can handle step 2 with a job that spawns/dispatches child jobs at runtime, but I do not know how to bind them properly to the existing and already running workflow.
Is it currently possible to achieve with Venture? If so, could someone give me a hint please?
Unfortunately, it currently isn't possible to dynamically add jobs to a running workflow. It's something I'm still working on solving.
Understood! 🥲
Is there any way to circumvent this limitation with a dirty hack in user land?
Not really, unfortunately.
The good news is that I think I just came up with the solution to this problem. I have a working prototype locally that let's you dynamically add jobs to a running workflow. I might be able to finish this up this weekend or next week 👍
OK got it, thank you for the quick answers!
Don't worry, no rush. Thank you for your work man. 🙂
@ksassnowski Thanks for your awesome project.
Just curious if you made any progress on this enhancement.
Ping! I am once again facing the situation where it would be much more convenient to do this. Does anyone knows?