until and while executing with different arguments for client/server
capsterx opened this issue · 2 comments
Is your feature request related to a problem? Please describe.
We have a worker process that updates a block of data, we were using :until_and_while_executing and it worked as expected. Recently we made an optimization to pass the worker arguments which allows it to limit what it looks at to update. We only want one process to run at a time so a workaround was to use just use while executing and ignore args.
Describe alternatives you've considered
I read the documentation about different args in server/client context.
- sidekiq unique only checks arguments client-side and they are restored from a hash server side
- Sidekiq::ProcessSet.new.size for my setup is the same on client and server.
Describe the solution you'd like
We want a lock like until executing that works normally and a lock while executing that limits to one per class and ignores arguments.
This is what I ended up implementing
module Locks
class UntilAndWhileExecutingServerArgs < SidekiqUniqueJobs::Lock::UntilAndWhileExecuting
include SidekiqUniqueJobs
def runtime_lock
new_item = item.dup
new_item[LOCK_ARGS] = Object.const_get(item[CLASS])
.sidekiq_options[LOCK_ARGS_METHOD]
.call(item[ARGS])
new_item[LOCK_DIGEST] = SidekiqUniqueJobs::LockDigest.call(new_item)
@runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(new_item, callback, redis_pool)
end
end
end
And for the worker
class Worker
include Sidekiq::Worker
sidekiq_options on_conflict: { server: :reschedule },
lock: :until_and_while_executing_server_args,
lock_args_method: lambda { |args|
if Sidekiq.server?
[]
else
args
end
}
end
And the config
SidekiqUniqueJobs.configure do |config|
config.add_lock :until_and_while_executing_server_args, Locks::UntilAndWhileExecutingServerArgs
end
Additional context
I wasn't sure if it would have been better to get server/client args both in the client side or allow it to run as I have in the sidekiq process. I also know this code is limited to a proc, I tried calling SidekiqUniqueJobs::LockArgs.call(new_item) but that didn't work for me for some reason, not sure if i was doing something wrong or not, but I went with just directly calling it (might have been an issue with me checking stuff in pry in sidekiq?). I also originally passed a ruby kw argument to the lambda until I found Sidekiq.server?. I assume that always works to check if it's running in sidekiq.
We had to roll this back as it was giving an error
gems/sidekiq-unique-jobs-7.1.8/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb:49 rescue in execute
ems/sidekiq-unique-jobs-7.1.8/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb:291 reflect
ArgumentError · wrong number of arguments (given 2, expected 0)
I had to change the code, one including SidekiqUniqueJobs got me variables, but imported a reflect that was not compatible. Also using Sidekiq.server? prevented reschedule from working because it was called from sidekiq
module Locks
class UntilAndWhileExecutingServerArgs < SidekiqUniqueJobs::Lock::UntilAndWhileExecuting
class RuntimeLock < SidekiqUniqueJobs::Lock::WhileExecuting
def runtime_lock
new_item = item.dup
new_item[SidekiqUniqueJobs::LOCK_ARGS] = Object.const_get(item[SidekiqUniqueJobs::CLASS])
.sidekiq_options[SidekiqUniqueJobs::LOCK_ARGS_METHOD]
.call(item[SidekiqUniqueJobs::ARGS], side: :server)
new_item[SidekiqUniqueJobs::LOCK_DIGEST] = SidekiqUniqueJobs::LockDigest.call(new_item)
@runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(new_item, callback, redis_pool)
end
end
end
class Worker
include Sidekiq::Worker
sidekiq_options on_conflict: { server: :reschedule },
lock: :until_and_while_executing_server_args,
lock_args_method: lambda { |args, side: :client|
if side == :server
[]
else
args
end
}
end