mhenrixon/sidekiq-unique-jobs

until and while executing with different arguments for client/server

capsterx opened this issue · 2 comments

Is your feature request related to a problem? Please describe.
We have a worker process that updates a block of data, we were using :until_and_while_executing and it worked as expected. Recently we made an optimization to pass the worker arguments which allows it to limit what it looks at to update. We only want one process to run at a time so a workaround was to use just use while executing and ignore args.

Describe alternatives you've considered
I read the documentation about different args in server/client context.

  1. sidekiq unique only checks arguments client-side and they are restored from a hash server side
  2. Sidekiq::ProcessSet.new.size for my setup is the same on client and server.

Describe the solution you'd like

We want a lock like until executing that works normally and a lock while executing that limits to one per class and ignores arguments.

This is what I ended up implementing
module Locks
  class UntilAndWhileExecutingServerArgs < SidekiqUniqueJobs::Lock::UntilAndWhileExecuting                 
    include SidekiqUniqueJobs                                                                              
    
    def runtime_lock
      new_item = item.dup
      new_item[LOCK_ARGS] = Object.const_get(item[CLASS])                                                  
        .sidekiq_options[LOCK_ARGS_METHOD]
        .call(item[ARGS])
      new_item[LOCK_DIGEST] = SidekiqUniqueJobs::LockDigest.call(new_item)                                 
      @runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(new_item, callback, redis_pool)        
    end             
  end                 
end   

And for the worker

class Worker
   include Sidekiq::Worker
   sidekiq_options on_conflict: { server: :reschedule },
                  lock: :until_and_while_executing_server_args,
                  lock_args_method: lambda { |args|
                    if Sidekiq.server?
                      []
                    else
                      args
                    end
                  }
end

And the config
SidekiqUniqueJobs.configure do |config|
config.add_lock :until_and_while_executing_server_args, Locks::UntilAndWhileExecutingServerArgs
end

Additional context
I wasn't sure if it would have been better to get server/client args both in the client side or allow it to run as I have in the sidekiq process. I also know this code is limited to a proc, I tried calling SidekiqUniqueJobs::LockArgs.call(new_item) but that didn't work for me for some reason, not sure if i was doing something wrong or not, but I went with just directly calling it (might have been an issue with me checking stuff in pry in sidekiq?). I also originally passed a ruby kw argument to the lambda until I found Sidekiq.server?. I assume that always works to check if it's running in sidekiq.

We had to roll this back as it was giving an error
gems/sidekiq-unique-jobs-7.1.8/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb:49 rescue in execute
ems/sidekiq-unique-jobs-7.1.8/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb:291 reflect
ArgumentError · wrong number of arguments (given 2, expected 0)

I had to change the code, one including SidekiqUniqueJobs got me variables, but imported a reflect that was not compatible. Also using Sidekiq.server? prevented reschedule from working because it was called from sidekiq

module Locks
  class UntilAndWhileExecutingServerArgs < SidekiqUniqueJobs::Lock::UntilAndWhileExecuting                 
    class RuntimeLock < SidekiqUniqueJobs::Lock::WhileExecuting                                            
    def runtime_lock
      new_item = item.dup
      new_item[SidekiqUniqueJobs::LOCK_ARGS] = Object.const_get(item[SidekiqUniqueJobs::CLASS])            
        .sidekiq_options[SidekiqUniqueJobs::LOCK_ARGS_METHOD]
        .call(item[SidekiqUniqueJobs::ARGS], side: :server)
      new_item[SidekiqUniqueJobs::LOCK_DIGEST] = SidekiqUniqueJobs::LockDigest.call(new_item)              
      @runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(new_item, callback, redis_pool)                                    
    end               
  end               
end  
class Worker
   include Sidekiq::Worker
   sidekiq_options on_conflict: { server: :reschedule },
                  lock: :until_and_while_executing_server_args,
                  lock_args_method: lambda { |args, side: :client|
                    if side == :server
                      []
                    else
                      args
                    end
                  }
end