ActiveRecord connection sharing: queue_classic cause rails application hang
Opened this issue · 2 comments
When ActiveRecord integration is enabled with QC_RAILS_DATABASE=true, queue_classic sometimes causes my rails application to hang, and I have to restart the application server to get it to work again. It seems to happen randomly, but is usually linked to an increase in activity. Unfortunately I haven't been able to reproduce the issue, it looks like a concurrency problem.
It looks like queue_classic was picking up a connection already used by Rails, causing a query to take a very long time (up to 20s) and fail. There was no specific increase in load on the database server, and it would happen with various query which were fine otherwise. The application would then hang, request were getting stuck to trying to get a database connection.
The problem went away when I switched to QC_DATABASE_URL. I suspect the issue is related to how queue_classic get a database connection out of the Rails connection pool, maybe related to :
317
Configuration :
ruby 2.4
Rails 4.2.11
queue_classic 3.1.0
Application is running on an amazon EC2 instance. Rails and QC are running on the same instance. QC is used to generated PDF.
@rioug are you using pooled connections? Did you get any further debugging this? I can't do much with your current details I'm afraid.
Yes I used pooled connections, it set to 33 on production.
I figured out that Rails was getting stuck in "ActiveRecord::QueryCache" middleware on subsequent request once I had query fail, that's as far as I got really. Unfortunately, I stopped looking into this once I found a work around. I though I'd report it anyway, in case someone run into a similar issue.
On way to maybe reproduce the issue, would be to set the connection pool to 1. Then have a script constantly sending request to the application and manually trigger a QueueClassic job ( or run a script to trigger a QueueClassic job every now and then).