Jobs lost during server restart
aclarembeau opened this issue · 3 comments
Hi Sidekiq Throttled team,
Thanks a lot for your awesome gem.
Recently, I spotted a problem on our sidekiq cluster, which caused some jobs to be lost when our servers were restarting (jobs were not picked up after restart, causing batch callbacks to never fire, and some other problems to happen).
I've opened a ticket on the official sidekiq repository (sidekiq/sidekiq#4607) and it seems that the problem can be caused by the fact that Sidekiq::Throttled (which we use) is replacing the superfetch strategy of sidekick.
Do you know is Sidekiq::Throttled can provide any guarantee that jobs will be picked up after the reboot of a Sidekiq node?
Thanks a lot
Did you find a solution or alternative for this? Where throttling works with super_fetch @aclarembeau
super-fetch support is planned, but as I don't have access to Sidekiq::Pro I can only guess how to better tackle that :D There was some work in that regard done:
- https://github.com/ixti/sidekiq-throttled/tree/sidekiq-pro-support
- https://github.com/ixti/sidekiq-throttled/tree/super-fetch
I'm working on cleaning up throttled fetch implementation to make integration with super fetch easier, but all I can do is to "guess" based on those two branches.