Concurrency misbehaving and causing duplicate jobs in queue

[danishsatkut]

I have been trying to get sidekiq-throttled (v1.0.0.alpha.1) to work with a new project (Rails v7.0.6 and Sidekiq v7.1.2) but I have been running into issues. When the sidekiq processing starts, the concurrency config is not respected. Once the first batch of 5 jobs (default sidekiq concurrency) are finished, the concurrency config starts working, but now it results in duplicate jobs (similar to #152).

I have attached the SampleJob and sidekiq.log which demonstrates the issue here: https://gist.github.com/danishsatkut/6a11d06eae2f7b464d285002856ba9b5

[danishsatkut]

So after playing around with different refs of the gem, I think the issue started somewhere after e5ac585. At that version, I wasn't seeing any duplication but the concurrency issue was still the same as I had encountered before. https://gist.github.com/danishsatkut/1c43401588f45368569499c3eae2982b

BTW I also downgraded sidekiq to v7.1.1

[emmahsax]

My team is using version 0.17.0, and we're seeing duplicate jobs as well.

[danishsatkut]

@emmahsax Which version of sidekiq are you using?

[emmahsax]

0.17.0. But I realized I didn't finish the docs. The README says the default lock is for 15 minutes, and that wasn't enough for our jobs. This is what I get for working at 5:30 in the morning 🤦🏼‍♀️ .

[ixti]

That's weird. Will debug.

[pentagon]

the same issue here, alpha seems not doing any concurrency management/throttling with the setup similar to the topic starter's one.

[ixti]

I have refactored the way BasicFetch is get patched. Should be good in >= v.1.0.0

[ixti]

This is the script I used to check:

require "sidekiq"
require "sidekiq/throttled"

puts "Sidekiq::VERSION            => #{Sidekiq::VERSION}"
puts "Sidekiq::Throttled::VERSION => #{Sidekiq::Throttled::VERSION}"

Sidekiq::Throttled.setup!

class SampleJob
  include Sidekiq::Job
  include Sidekiq::Throttled::Job

  sidekiq_throttle concurrency: { limit: 1 }

  def perform(id)
    puts "Processing ##{id}"
    sleep(10)
    puts "Done ##{id}"
  end
end

100.times { SampleJob.perform_async(_1.succ) }

Output

> be sidekiq -r ./app.rb
Sidekiq::VERSION            => 7.2.0
Sidekiq::Throttled::VERSION => 1.0.1
2023-11-20T16:58:51.483Z pid=3049 tid=2f5 INFO: Sidekiq 7.2.0 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>nil}


               m,
               `$b
          .ss,  $$:         .,d$
          `$$P,d$P'    .,md$P"'
           ,$$$$$b/md$$$P^'
         .d$$$$$$/$$$P'
         $$^' `"/$$$'       ____  _     _      _    _
         $:    ',$$:       / ___|(_) __| | ___| | _(_) __ _
         `b     :$$        \___ \| |/ _` |/ _ \ |/ / |/ _` |
                $$:         ___) | | (_| |  __/   <| | (_| |
                $$         |____/|_|\__,_|\___|_|\_\_|\__, |
              .d$$                                       |_|
      

2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: Running in ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: Sidekiq 7.2.0 connecting to Redis with options {:size=>5, :pool_name=>"default", :url=>nil}
2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: Starting processing, hit Ctrl-C to stop
2023-11-20T16:58:51.500Z pid=3049 tid=qd class=SampleJob jid=00cc61231cb117f81dd31853 INFO: start
Processing #1
Done #1
2023-11-20T16:59:01.502Z pid=3049 tid=qd class=SampleJob jid=00cc61231cb117f81dd31853 elapsed=10.002 INFO: done
2023-11-20T16:59:01.511Z pid=3049 tid=pl class=SampleJob jid=48bec13f40aacc1fb3946018 INFO: start
Processing #14
Done #14
2023-11-20T16:59:11.512Z pid=3049 tid=pl class=SampleJob jid=48bec13f40aacc1fb3946018 elapsed=10.002 INFO: done
2023-11-20T16:59:11.519Z pid=3049 tid=od class=SampleJob jid=3063d342d29da264b4b07ac3 INFO: start
Processing #20
Done #20
2023-11-20T16:59:21.519Z pid=3049 tid=od class=SampleJob jid=3063d342d29da264b4b07ac3 elapsed=10.0 INFO: done
2023-11-20T16:59:23.520Z pid=3049 tid=pl class=SampleJob jid=68cc2c6c29f92560be4165e4 INFO: start
Processing #37
Done #37
2023-11-20T16:59:33.521Z pid=3049 tid=pl class=SampleJob jid=68cc2c6c29f92560be4165e4 elapsed=10.001 INFO: done
2023-11-20T16:59:33.535Z pid=3049 tid=oh class=SampleJob jid=7823134f81d36c20ff8fef0c INFO: start
Processing #54
Done #54
2023-11-20T16:59:43.536Z pid=3049 tid=oh class=SampleJob jid=7823134f81d36c20ff8fef0c elapsed=10.001 INFO: done
2023-11-20T16:59:43.545Z pid=3049 tid=qd class=SampleJob jid=7db02baafd2b97f58da996b2 INFO: start
Processing #65
Done #65
2023-11-20T16:59:53.546Z pid=3049 tid=qd class=SampleJob jid=7db02baafd2b97f58da996b2 elapsed=10.001 INFO: done
2023-11-20T16:59:53.552Z pid=3049 tid=od class=SampleJob jid=019a13a53200fcfdf55e3c9b INFO: start
Processing #71
Done #71
2023-11-20T17:00:03.553Z pid=3049 tid=od class=SampleJob jid=019a13a53200fcfdf55e3c9b elapsed=10.001 INFO: done
2023-11-20T17:00:05.534Z pid=3049 tid=n9 class=SampleJob jid=e4b16d0ffde0ebf5df5bc108 INFO: start
Processing #82
Done #82
2023-11-20T17:00:15.535Z pid=3049 tid=n9 class=SampleJob jid=e4b16d0ffde0ebf5df5bc108 elapsed=10.001 INFO: done
...

[danishsatkut]

@ixti I will test and confirm if it works. Thank you for your efforts.

[danishsatkut]

@ixti I have tested and can confirm the issue is resolved after upgrading to v1.0.1 (#166)

[ixti]

@danishsatkut thank you!