Force pipeline termination from pipeline manager
ryzhyk opened this issue · 2 comments
ryzhyk commented
The runner handles the /shutdown command by first calling controller.stop() and then sending a termination message to the actix dispatch loop. Both operations can be slow or even get stuck in some cases. We will need a way to enforce eventual termination from the pipeline manager by killing the process/pod after a timeout.
lalithsuresh commented
@ryzhyk I get the operations being slow, but do we know when/why they get stuck? Because that would be a bug.
ryzhyk commented
controller.stop()waits for any in-progress iteration of the circuit to complete. Which should normally be fast, but can take a while if there's a perf issue in the circuit or we're processing a very large input batch. Another possible reason is buggy connectors that refuse to shutdown instantly or at all.- actix is not supposed to take long, but I think I've seen it take a while to shutdown. Not sure what's up with that.