After sidekiq shutdown (with running job) lock digest is not pushned back
kirsha2 opened this issue · 8 comments
Describe the bug
Rails: 7.0.4.3
Ruby: 3.2.2
Sidekiq: 6.2.1
When trying upgrading sidekiq and unique jobs our tests began to fail. Same behavior in manual testing.
- Enqueue until_executed job
- start processing
- shutdown worker (job is still not finished)
- job is pushed back into queue but not the digest
The bug occurs after 7.1.29 version. Same behavior in 8.
Used lock: until_executed
Expected behavior
After sidekiq shutdown job and digest are pushed back.
Current behavior
After sidekiq shutdown only job is pushed back.
I've looked into diff between version 29 and 30 but I cant see anything suspicious there. Is this expected behavior ?
Thanks :) for all your hard work - love the gem!
I've tried to look into the changes more and found out it was due to the ensure in until_executed.rb
https://my.diffend.io/gems/sidekiq-unique-jobs/7.1.29/7.1.30#d2h-630681
So I don't know if this change/ consequence was intentional or not
I've tried to look into the changes more and found out it was due to the ensure in until_executed.rb
https://my.diffend.io/gems/sidekiq-unique-jobs/7.1.29/7.1.30#d2h-630681
So I don't know if this change/ consequence was intentional or not
It sounds like a bug; I'll see about fixing these issues in the next few days. I should use an else rather than an ensure.
what behavior does this cause? Currently i'm facing an issue where my until_executed jobs can't get enqueued anymore and this sounds like it could be related to my problem. if so Is there something i can run from the console that will fix this for me when this happens?
@joshuacronemeyer sounds like your locks/jobs are not correctly unlocked.
Behavior that I reproduced only affects re-enqued jobs, so if the job is not finished but pushed back into to queue.
This is issue is about missing lock and yours is probably still there "stuck".
I'd recommend to check your Unique Digests.
@kirsha2 The ensure-based bug was fixed in v8.0.5 and v7.1.31 I think. thread: #800 (comment)
Unfortunally I closed it prematurely, I've forgotten to remove my monkey patch in the code (too many changes in MR :D )
The main issue for me is still the ensure part in
My expected behavior, when job is in retry set the lock is still kept.
When this job after retries goes to dead, death_handlers takes care of unlocking
https://github.com/mhenrixon/sidekiq-unique-jobs/blob/main/lib/sidekiq_unique_jobs/server.rb#L13
I patched it like this
# lib/sidekiq_unique_jobs/lock/until_executed.rb
def execute
executed = locksmith.execute do
yield
unlock_and_callback # NOTE: override ensure due to lock not pushing back after sidekiq shutdown
end
reflect(:execution_failed, item) unless executed
nil
end