until_and_while_executing with separate server and client lock_args broken since 8.0.8
Roguelazer opened this issue · 7 comments
Describe the bug
After upgrading from 8.0.7 to 8.0.10, jobs with until_and_while_executing
that pass a custom lock_args_method
which behaves differently on the server and the client (as shown in the README) no longer work. Based on bisection, this appears to have been broken by 63e9431 (the parent commit, 7d5e40a, still works).
Expected behavior
Given a worker as shown below, run the following client code:
1.upto(2)
UniqueBugWorker.perform_async(id)
UniqueBugWorker.perform_async(id)
UniqueBugWorker.perform_async(id)
UniqueBugWorker.perform_async(id, "foo")
UniqueBugWorker.perform_async(id, "foo", "bar")
end
We expect to see exactly six invocations.
Current behavior
Since 63e9431, we end up seeing at most two invocations (and sometimes only 1, which is even more worrying). It appears that the lock digest is being reused between the until and while executing phases.
For some reason, the conflicted jobs also aren't getting rescheduled; they just get dropped on the floor.
worse yet, we end up creating and leaking uniquejobs: locks with the correct digest, so after downgrading to 8.0.7, the jobs are locked out until the lock TTL expires.
Worker class
class UniqueBugWorker
include Sidekiq::Worker
sidekiq_options queue: "repro",
lock: :until_and_while_executing,
on_conflict: {server: :reschedule, client: :log},
lock_args_method: ->(args) do
if Sidekiq.server?
[args.first]
else
args
end
end
def perform(id, foo = nil, bar = nil)
Rails.logger.info { "UniqueBugWorker: performing #{id} with #{foo} #{bar}" }
sleep 1
end
end
Additional context
Add any other context about the problem here.
We seem to have been running into same issues since upgrade but with until_executed
(possibly with other locks as well but I only have confirmation about issues with until_executed
)
@reneklacan do you also use custom lock args?
@mhenrixon no