Concurrency limits entire queue, not only the job type

[peplin]

We're experiencing an odd behavior when using the concurrency limiter on a job. It appears to be limiting the concurrency of the queue that the job is running in, and not only the job with the concurrency limit.

Here is a small example:

class ConcurrencyLimitedJob
   ...
   sidekiq_throttle(concurrency: {limit: 8})
   sidekiq_options queue: "low"
end

class UnlimitedJob
   ...
   sidekiq_options queue: "low"
end

We run 2 Sidekiq worker processes, with 10 threads each.

What we expect is that if there are 1000 instances of ConcurrencyLimitedWorker in the queue, only 8 will ever be running at one time. If we enqueue an instance of UnlimitedJob at any time, it should run immediately because there are 12 free Sidekiq worker threads.

What we are seeing instead is that the UnlimitedJob will only run after all 1000 instances of ConcurrencyLimitedWorker finish. Effectively, the concurrency limit of 8 on that job has limited the concurrency of the entire queue.

If I replace the concurrency strategy with a threshold, the issue does not appear. The UnlimitedJob is executed immediately, even if there are still more of the throttled jobs in the queue.

Is it possible we have something misconfigured? Thanks!

[peplin]

Today, we're now seeing similar behavior with threshold limiters as well. I couldn't reproduce it the small test case described above, but in our production setting we're seeing only 3/20 Sidekiq threads being used while there are 20k supposedly unthrottled jobs waiting in the queue.

Interestingly, I found that if I clicked the "Reset" button for one of the jobs being rate limited, suddenly all 20 threads were busy and we started burning through the queue of jobs rapidly.

[payuramex]

We're also getting this exact same behavior. Anyone have a fix?

[ixti]

That is, somewhat, regression that is caused by an attempt to reduce the load on redis when queue has only throttled jobs. When fetch retrieves a job that is throttled it pushes it back to the queue and skips fetching that queue for next 2 seconds (by default). You can change this behavior by setting throttled_queue_cooldown option:

# Sidekiq 6.4
Sidekiq.options[:throttled_queue_cooldown] = 0

# Sidekiq 6.5
Sidekiq[:throttled_queue_cooldown] = 0

# Sidekiq 7.0
Sidekiq.default_configuration[:throttled_queue_cooldown] = 0

[ixti]

Related to #94

[ixti]

I have removed the queue cooldowns in v1.0.0.alpha;
Later will introduce an option to bring them back based on specific needs (dynamic queues filtering + observers)