rails/solid_queue

Discard duplicate jobs

Opened this issue · 4 comments

matteeyah commented

Summary

The concurrency controls that currently exist allow blocking execution of "duplicate" jobs, or jobs with the same arguments. These jobs wait, then execute after the currently running job finishes. There's no way to discard all duplicate jobs and prevent them from running completely.

Proposal

Add a way to discard jobs that get blocked by the concurrency controls.

rosa commented

@matteeyah, yes! I've got this on my mind as well! In the end, I'm not sure there's a version of #105 that I like or that it's useful for what we need, so I thought, in the meantime, to add another parameter to limits_concurrency to discard instead of blocking. Something like:

on_conflict: :discard

By default, it'd be:

on_conflict: :block

which would be the current behaviour.

  • 👍2
matteeyah commented

That's exactly what I had in mind!

For reference - We're currently using https://github.com/veeqo/activejob-uniqueness. It serves a slightly different purpose, but it also checks for duplicate jobs.

It also has an on_conflict option - it supports logging a conflict, raising an error on a conflict and passing a proc to handle the conflict manually.

  • 👀1
nhorton commented

Another vote for following the activejob-uniqueness conventions.

I would note that of the strategies offered by the gem, until_and_while_executing is the one we have found most-useful.

  • 👍1
nhorton commented

Is there any more word on this? Or any openness to others trying to contribute?