que-rb/que

Multiple job are not executed in parallel

dima4p opened this issue · 3 comments

I stat que runner with que -w 6 -l debug &
Then I add 5 similar jobs that should start in a minute.
When the time comes only one job starts.
While it runs select("id, pg_try_advisory_lock(#{que_job.id}) as taken") for every job even of different job_class shows that the job is locked.
After the running job is completed the rest 4 are released. And it seems they run in parallel.
Though now the time window is closed already and all jobs terminate immediately due to the logic.
The difference is the first job calls a method that is running almost completely within a transaction. Others do not open a transaction.
1.0.0.beta4

That's odd!

What versions of Postgres and Ruby?

Do the other jobs start executing at all?

Does it also happen if you run multiple que processes?

Does your database connection pool have enough capacity?

What versions of Postgres and Ruby?

postgresql-12.3
ruby '2.7.1'

Do the other jobs start executing at all?

With other jobs there is no problem. Though, they are not being executed in parallel

Does it also happen if you run multiple que processes?

Should I? I thought one master process is enough and it will others start.

Does your database connection pool have enough capacity?

Yes, max_connections = 100

The code is like this:

  def perform(*args)
    @end_at = Time.parse Setting.time_parsing_end
    while Time.current < @end_at
      Log.transaction do
        log = Log.lock('FOR UPDATE SKIP LOCKED').unparsed.first
        return unless log
        log.create_lines!
      end
      @end_at = Time.parse Setting.time_parsing_end
    end
  end

Last time it seems not to block other DJs. I close the issue until meet the case again.