One consumer only for specific queue
TarasBardiuk opened this issue · 0 comments
Hi, guys.
I need to fetch and process messages from one specific queue in a strict stream, messages should be processed one by one, I mean NOT in parallel. We need this to avoid parallel creation of records, only one by one.
I was trying to achieve that by local worker config. As a result, thread: 1
doesn't help because I still have 4 consumers (workers: 4
specified in a config yml file). Then I was trying to use local workers
option in a specific worker class, but it doesn't work, I guess it is not supported at all.
Then I found Handling different workloads Wiki-page, but it didn't help as well.
- I configured
./config/sneaker_worker_groups.yml
file:
notparallelgroup:
classes: MyWorkerClass
workers: 1
- Then I executed
bundle exec rake sneakers:run
task
After this step, I assumed that now all my queues should be handled by 4 consumers (as by default), but one specific queue (managed byMyWorkerClass
) by a single consumer only. But no, all queues where listened by 4 consumers. Okay, maybe I need to run spawner manually? - So I executed
bundle exec ruby -e "require 'sneakers/spawner';Sneakers::Spawner.spawn"
as mentioned in a Wiki page and... nothing changed. Every queue was still listened by 4 consumers.
Then I tried to create my own rake task to run it instead of default sneakers:run
:
require 'sneakers'
require 'sneakers/runner'
task :environment
namespace :sneakers do
desc 'Custom sneakers process start'
task start: :environment do
::Sneakers.server = true
::Rake::Task['environment'].invoke
::Rails.application.eager_load!
default, specific = ::Sneakers::Worker::Classes.partition do |classy|
!classy.queue_opts.key?(:workers) || classy.queue_opts[:workers] ==
::Sneakers.const_get(:CONFIG)[:workers]
end
::Sneakers::Runner.new(default, {}).run
specific = specific.group_by { |classy| classy.queue_opts[:workers] }
specific.each do |workers, classes|
::Sneakers::Runner.new(classes, workers: workers).run
end
end
end
But I found it impossible to be a single task only, because after the first #run
method call, the rest of the rake task is not executed at all. So in such case I will have to define separate rake tasks for each local workers
config option value.
It may be worth to say that on a server I need to start sneakers in a daemonize: false
mode, but locally it is always daemonize: true
.
Please help me get one specific queue to be processed by a single consumer only.
P.S. Why don't worker groups work? Or am I wrong and they are useless for my needs?
Thank you in advance.