What happens with Sneakers job that running when the worker node is restarted?
paxer opened this issue · 3 comments
I can't find the answer in the wiki, and I need to compare with Sidekiq behaviour which is:
By default, Sidekiq gives workers 8 seconds to shut down. This is carefully chosen because Heroku gives a process 10 seconds to shutdown before killing it. After 8 seconds, any remaining jobs still in progress are pushed back onto Redis so they can be immediately restarted when Sidekiq starts back up. Remember that Sidekiq will run your jobs AT LEAST once so they need to be idempotent. This is one example of how a job can be run twice.
how Sneakers will handle this situation?
I think I found the answer in RabbitMQ docs...
Doing a task can take a few seconds. You may wonder what happens if one of the consumers starts a long task and dies with it only partly done. With our current code, once RabbitMQ delivers a message to the consumer it immediately marks it for deletion. In this case, if you kill a worker we will lose the message it was just processing. We'll also lose all the messages that were dispatched to this particular worker but were not yet handled.
But we don't want to lose any tasks. If a worker dies, we'd like the task to be delivered to another worker.
In order to make sure a message is never lost, RabbitMQ supports message acknowledgments. An ack(nowledgement) is sent back by the consumer to tell RabbitMQ that a particular message has been received, processed and that RabbitMQ is free to delete it.
If a consumer dies (its channel is closed, connection is closed, or TCP connection is lost) without sending an ack, RabbitMQ will understand that a message wasn't processed fully and will re-queue it. If there are other consumers online at the same time, it will then quickly redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally die.
There aren't any message timeouts; RabbitMQ will redeliver the message when the consumer dies. It's fine even if processing a message takes a very, very long time.
one thing I am still a bit confused though, probably because of my yet limited knowledge about RabbitMQ... Why Sneakers need Redis if RabbitMQ already storing not delivered messages and queues? 🤔
ah I was mislead by the repo Readme example, Sneakers does not require Redis 👍