Jobs disappear from the queue if the number of retry attempts is exceeded

[squaretus]

Good afternoon, I have a question related to configuration.
I'm using Ruby 3.2.2, Rails 7.0.8, Sidekiq-Throttled 1.0.0.alpha

The task was as follows:
It is necessary to configure the job execution delay so that jobs with the same Integration_id are executed with a delay between them in an arbitrary period of time from 61 to 97 seconds.
Jobs with different Integration_ids could be executed quietly without delay using standard concurrency (up to 10 jobs simultaneously)

If there is no integration_id argument, then there is no need to delay job execution at all and we execute all jobs in the queue using standard concurrency (up to 10 at a time)

My setup now looks like this

  sidekiq_throttle(
    concurrency: {
      limit: ->(_, ids) { ids['integration_id'].present? ? 1 : 10 },
      key_suffix: ->(_, ids) { ids['integration_id'].present? ? ids['integration_id'] : nil }
    },
    threshold: {
      limit: 1,
      key_suffix: ->(_, ids) { ids['integration_id'].present? ? ids['integration_id'] : nil }
    }
  )

in sidekiq initializer it is specified by gem only

require 'sidekiq/throttled'

Sidekiq::Throttled.setup!

At first glance, everything works correctly, but I ran into the problem that jobs that cannot be executed at the moment go into retry, which by default are 25
The execution time of the job is very low, therefore, about 25 seconds pass and the job is simply killed, since the window of 61 - 97 seconds has not yet passed for the possibility of launching a new job with the same integration_id that has already been launched or is currently running

Are there solutions for this problem and have I configured everything correctly based on the current task?
or should you wait for a #150?

Could a solution be to remove a queue from the pool that has at least one throttled job?

Sidekiq::Throttled.configure do |config|
  # Period in seconds to exclude queue from polling in case it returned
  # {config.cooldown_threshold} amount of throttled jobs in a row.
  config.cooldown_period = 79.0

  # Exclude queue from polling after it returned given amount of throttled
  # jobs in a row.
  config.cooldown_threshold = 1
end

[ixti]

Hi! Cooldown was (re-)introduced in sidekiq-throttled 1.0.0 (it's not available in 1.0.0.alpha). Jobs are going to retry when they fail. I believe 1.0.0.alpha had a bug where throttling was not working at all. Can you please check with sidekiq-throttled 1.0.0?

[ixti]

Also, try changing the throttling config to look like:

  sidekiq_throttle(
    concurrency: {
      limit: ->(_, ids) { ids['integration_id'].present? ? 1 : 10 },
      key_suffix: ->(_, ids) { ids['integration_id'].presence || 'default' }
    },
    threshold: {
      limit: 1,
      key_suffix: ->(_, ids) { ids['integration_id'].presence || 'default' }
    }
  )

[squaretus]

Regarding the first message - that’s exactly what I did yesterday, everything worked, thank you!
We need to be careful when using dependabot, which sometimes posts weird versions of gems like 1.0.0.alpha...

Regarding the second message - thank you for the proposed edits, I will add them to the list of tasks for today and in today's release!

I was very glad to quickly receive help on the issue and wish you success in the development of your gem!

Looking forward to 2.0.0!