Changing the throttling config when jobs are running
roykolak opened this issue · 0 comments
roykolak commented
I'm currently seeing batches of jobs that are configured to throttle dynamically get stuck when the throttle configuration changes.
For example, here's my throttle config...
sidekiq_throttle(
concurrency: {
limit: ->(search_id, result_id, batch_id, dynamically_assigned_queue) {
Search.slow_search?(search_id, batch_id) ? 1 : 5
},
key_suffix: ->(search_id, result_id, batch_id, dynamically_assigned_queue) {
Search.slow_search?(search_id, batch_id) ? "#{dynamically_assigned_queue}-slow" : "#{dynamically_assigned_queue}-fast"
}
},
observer: ->(strategy, *args) {
Sidekiq.logger.warn "Throttled: #{strategy} search_id: #{args[0]} queue: #{args[3]}"
}
)
You can see that the jobs belong to a batch and that batch gets assigned to a random shared queue.
My question is, if Search.slow_search?
returns a different value in the middle of running a batch of jobs, is that a problem? I'm seeing a lot of logging that the "concurrency" strategy prevented jobs from running after Search.slow_search?
changes it's boolean value.