Multiple job are not executed in parallel
dima4p opened this issue · 3 comments
I stat que runner with que -w 6 -l debug &
Then I add 5 similar jobs that should start in a minute.
When the time comes only one job starts.
While it runs select("id, pg_try_advisory_lock(#{que_job.id}) as taken")
for every job even of different job_class
shows that the job is locked.
After the running job is completed the rest 4 are released. And it seems they run in parallel.
Though now the time window is closed already and all jobs terminate immediately due to the logic.
The difference is the first job calls a method that is running almost completely within a transaction. Others do not open a transaction.
1.0.0.beta4
That's odd!
What versions of Postgres and Ruby?
Do the other jobs start executing at all?
Does it also happen if you run multiple que
processes?
Does your database connection pool have enough capacity?
What versions of Postgres and Ruby?
postgresql-12.3
ruby '2.7.1'
Do the other jobs start executing at all?
With other jobs there is no problem. Though, they are not being executed in parallel
Does it also happen if you run multiple
que
processes?
Should I? I thought one master process is enough and it will others start.
Does your database connection pool have enough capacity?
Yes, max_connections = 100
The code is like this:
def perform(*args)
@end_at = Time.parse Setting.time_parsing_end
while Time.current < @end_at
Log.transaction do
log = Log.lock('FOR UPDATE SKIP LOCKED').unparsed.first
return unless log
log.create_lines!
end
@end_at = Time.parse Setting.time_parsing_end
end
end
Last time it seems not to block other DJs. I close the issue until meet the case again.