ixti/sidekiq-throttled

Observer is called continuously

stefanoc opened this issue · 4 comments

stefanoc commented

Since I need to upgrade Sidekiq to version 7, I'm testing the 1.x branch (I've been using version 0.18 so far). Everything looks fine except for one thing: the observer callback is invoked much more frequently than before. I'm talking about hundreds of calls per second.
This is my test job:

class TestJob < ApplicationJob
  include Sidekiq::Throttled::Job

  OBSERVER = lambda do |strategy, *args|
    puts "THROTTLING: #{strategy} #{args.inspect}"
  end

  sidekiq_throttle(
    concurrency: { limit: 10 },
    threshold: { limit: 10, period: 1.hour },
    observer: OBSERVER
  )

  def perform
    sleep 1
  end
end

Queueing 20 jobs:

 20.times { |n| TestJob.perform_async }

and letting Sidekiq run for 2-3 seconds results in >2000 log lines printed in the terminal.
Is this expected behavior?

ixti commented

I've removed expireable list that was casuing issues for one group of users. But removing it - causes now issues for the other group. So, I'm planning on brining it back with more obvious ways to configure the behaviour.

  • 👍1
akarimcheese commented

I've removed expireable list that was casuing issues for one group of users. But removing it - causes now issues for the other group. So, I'm planning on brining it back with more obvious ways to configure the behaviour.

In bringing this back, would this include the behavior of fetching from lower queues while a higher one is in the expirable list?

ixti commented

Yeah, expirable set will be responsible to exclude queues that got too many throttled jobs in a row for a fetcher for the duration of configurable time.

PS: The work has began, thanks @Connorhd: #160

ixti commented

Will merge the fix today!

  • 🎉1