How to offload long running task
Closed this issue · 2 comments
In some of my routes I need to start a long running CPU intensive task. For example generating a ZIP file, and then the user can ask another endpoint for the status. If I start the task within a route, and the user disconnects, then the thread (aka the task) is cancelled.
How can this be handled and ensuring, that the thread releases the memory afterwards?
I can easily spawn a new thread with createThread, but what would be the best practices for joining it afterwards? Or could I force mummy to ignore the client and use the workerthread to fulfill the job ignoring the disconnect?
My suggestion here somewhat depends on a couple dimensions. Is this a rare or frequent task being started? Is this ZIP creation a potentially very large file or always small?
There is a threshold of infrequent + small where I think just having a fixed size queue of threads on the server would work, and I'd just ignore the disconnecting as it probably doesn't matter.
If this could be a frequent task or large, I'd split this and have the API server just start the task by adding it into a Redis queue of some kind, and have a separate one or more processes / machines that just pull from that queue and complete that task. This way I can have any number of tasks running in parallel completely independent from the API server.
I prefer not mixing API requests which are fast and light with these long running things so I have done and prefer the simple Redis thing. But for small enough things or if its just not a big deal, then a little task queue locally on the machine would be fine (or even just blocking the Mummy thread).
Also, Redis is just an example, any shared db could be fine.
Either way I would probably not be doing createThread in an arbitrary user-accessible endpoint, nor worry about joining threads later etc since a fixed pool is more robust etc. The queue can just have threads sitting idle at zero cost until work comes in.
Thank you for the insights guzba, and I agree on the concepts. I'll try some approaches using channels. Thanks!