PG::ConnectionBadqc:work PQconsumeInput() connection not open
udfalkso opened this issue ยท 13 comments
How can I fix this issue? It's happening in production but not development. There is an issue from 2013 that seems relevant, but slightly different. My setup with rails is totally straightforward, no shared connections, etc. Any ideas would be appreciated. Thanks!
bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:62:in `wait_for_notify': PQconsumeInput() connection not open
(PG::ConnectionBad)
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:62:in `block in wait_for_notify'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:61:in `tap'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:61:in `wait_for_notify'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:34:in `block in wait'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:31:in `synchronize'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:31:in `wait'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/worker.rb:96:in `lock_job'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/worker.rb:72:in `work'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/worker.rb:46:in `start'
from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/tasks.rb:27:in `block (2 levels) in <main>'
from gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
from gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
from gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
from gems/bugsnag-6.11.1/lib/bugsnag/integrations/rake.rb:18:in `execute_with_bugsnag'
from gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
from /usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
from gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
from gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
from gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
from gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
from gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
from gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
from gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
from gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
from gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
from gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
from gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
from gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
from bin/rake:23:in `load'
from bin/rake:23:in `<top (required)>'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `load'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `kernel_load'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:28:in `run'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:463:in `exec'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:27:in `dispatch'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:18:in `start'
from /usr/local/bin/bundle:30:in `block in <main>'
from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
from /usr/local/bin/bundle:22:in `<main>'
I'm running into an issue where PG::ConnectionBad: connection is closed
happens. I recently upgraded to rails 5.2 which seems to be the issue here.
Do you use rails or just plain queue_classic? If so what version? Do you use the automatic connection sharing, or did you disable it by setting QC_RAILS_DATABASE=false
?
I think to help resolve this, the maintainers will need a lot more information that just stacktrace
I'm running into an issue where
PG::ConnectionBad: connection is closed
happens. I recently upgraded to rails 5.2 which seems to be the issue here.
...
I run into the same issue currently and have no idea how to fix it besides fiddling with queue_classic
internas. Happened after I've upgraded our application from rails 4.x to 5.2. It seems to only happen from time to time, non-deterministic, and also just in production but not in development.
@vongruenigen Use QC_RAILS_DATABASE=true
, fixed it for me..
I was able to reproduce it, when a high load appeared on my web site. After a while of lower load, the PG::ConnectionBad
appeared. (Probably some garbage collector cleaning up AR connections without queue classic noticing it)
@siegy22 Thanks, I'll immediately try QC_RAILS_DATABASE=true
. I also have the problem that the problem occurs with really low load, namely only me being the only user manually testing stuff on our app on a test server.
We'll probably switch to specifying the database via QC_DATABASE_URL
in the long run, but I'll have to wait until one of our DevOps show up to delegate this task
@vongruenigen You can also try out my fork, I've a applied a small patch. Which should make the connection sharing with active record work again.
@siegy22, thanks, I'll probably try it later somewhen
@siegy22, have you tried setting the ENV['QC_DATABASE_URL']
directly in application.rb
? Like so:
if ENV['QC_RAILS_DATABASE'].blank? && ENV['QC_DATABASE_URL'].blank?
db_conf = config.database_configuration[Rails.env].symbolize_keys
db_url = "#{db_conf[:adapter]}://#{db_conf[:username]}"
# more string fiddling ...
ENV['QC_DATABASE_URL'] = db_url.freeze
ENV['QC_RAILS_DATABASE'] = false.to_s
end
This seems to work for me too, but I'm not happy with building the database URL by concatenating strings tbh.
Yeah I've tried several ways to do what you're attempting to do, but always found a new way which made it fail and ended up running from my fork for now.
I'm trying to get this fixed upstream, and maybe even get a release
@siegy22 I'm checking out your code ^ -- looks like it might be backwards incompatible?
@siegy22 this might be due to connections taken with ActiveRecord::Base.connection
are released to the pool by Action Pack's request handling cycle. To lease connection properly (according to AR docs), ActiveRecord::Base.connection_pool.checkout
should be used.
I opened #317, as it looks distinct from the problem described initially here: PQconsumeInput()
might be caused just by TCP connection closed or other causes, as opposed to PG::ConnectionBad: connection is closed
which is caused only by .close
being called on connection object: this closed state is distinct from disconnect and can't be recovered by reset
. (Pool might close connections it in e.g. flush).
Any chances rails/rails#36473 (released 2 days ago as part of Rails 2.5.4) fixed the problem?