Abort/Kill jobs over limit?

[9mm]

I want to use this gem for 'email warmup' for an email sending IP.

It's basically X sends per hour, gradually increasing each day. Our email volume is much greater than the warmup schedule, so its likely that many emails will be 'on hold' for potentially days, until the warmup schedule meets our volume.

The thing is, if the job is say, older than 1 hour, I want to literally just kill the job and not perform it at all. So I want the exact functionality of throttling, but with "over X time period, kill job"... this is rather than just keeping jobs for say an entire week THEN sending it which could lead to higher spam rates (getting a welcome email 6 days later for instance)

Is there a way I can do that with this gem?

[9mm]

The closest thing I have got is I pass a Time.zone.now into the job when its created as an argument.

I've messed around with an observer, and I can parse the time and see if it's older than Time.zone.now - 1.hour, the problem is I want to just delete the job right there.

I've done the check in the job itself (in the perform method) but that also doesn't work obviously as jobs which are deleted should not count in the limit, so simply returning early from the job is no good.

I would imagine in the observer I need to somehow delete the job if some condition is true, otherwise do nothing

[ixti]

Currently, I believe what you described with created_at is the best option. You can use dynamic limit and return false-ish value to signal that throttling should not be done for the job:

class SomeWorker
  # ...

  TOO_OLD = ->(options) { Time.at(options.fetch("created_at")) < 1.hour.ago }

  sidekiq_throttle({
    :threshold => {
      :limit  => proc { |options| TOO_OLD[options] ? false : 10 },
      :period => 1.minute
    }
  })

  def perform(options)
    return logger.warn("too old") if TOO_OLD[options]
  end
end

In the future we can enhance API of throttling rules to support something like :exclude => rules, and in general, I believe all these procs should have access to underlying job message, as it already contains created_at timestamp. But it's not right now.

[9mm]

awesome, thanks!