Queue system for tasks
hsuperman opened this issue · 5 comments
It would help if there was a queue system where the tasks were completed sequentially, and/or there is some limit to the number of tasks being executed in parallel.
Currently, the system tries to execute all tasks immediately, which can cause the machine to crash. A queue, combined with more clear statuses around tasks, would be extremely helpful.
@hsuperman yes, totally agree.
Skyvern open source is actually built with fastapi background task but somehow there's a bug blocking it from running parallel tasks. Will prioritize on it later this week.
We've built such a distributed system with aws sqs + aws batch in Skyvern Cloud. The work has not been prioritized in the open source side but definitely in our roadmap. We're thinking of migrating to a more open source driven solution. We're thinking about https://hatchet.run/ or https://temporal.io/ as the task queue solution. Open to suggestions
Without knowing much about your system, I would try to implement a simple queue using postgres since it's already there? Even a rudimentary queue through adding a few columns to the existing tasks table might be good enough..
…
On Wed, Jul 3, 2024, 6:21 PM Shuchang Zheng @.> wrote: @hsuperman https://github.com/hsuperman yes, totally agree. Skyvern open source is actually built with fastapi background task https://fastapi.tiangolo.com/tutorial/background-tasks/ but somehow there's a bug blocking it from running parallel tasks <#545>. Will prioritize on it later this week. We've built such a distributed system with aws sqs + aws batch in Skyvern Cloud. The work has not been prioritized in the open source side but definitely in our roadmap. We're thinking of migrating to a more open source driven solution. We're thinking about https://hatchet.run/ or https://temporal.io/ as the task queue solution. Open for suggestions — Reply to this email directly, view it on GitHub <#488 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJSVJZ7NICFN5KOGSIR2ALZKR2O7AVCNFSM6AAAAABJQPMQ3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBXGQYDGMRZGY . You are receiving this because you were mentioned.Message ID: @.>
Got it. Thanks for the suggestion. It can be a good potential alternative for the existing fastapi background task.
We'll think about it more. For us, we really want to build something that benefits the open source community, as well as solving problems on the cloud/enterprise side (multi tenant, large scale task queue)
This issue is stale because it has been open for 30 days with no activity.