PQsocket() can't get socket descriptor (PG::ConnectionBad) locker error
k1r8r0wn opened this issue · 3 comments
Hey there, I'm periodically getting this:
Que 2.3.0 started worker process with: Worker threads: 2 (priorities: any, any) Buffer size: 8 Queues: - default (poll interval: 5s)Que waiting for jobs...#<Thread:0x00007fcbdf582598 /usr/src/app/vendor/cache/ruby/3.2.0/gems/que-2.3.0/lib/que/locker.rb:179 run> terminated with exception (report_on_exception is true):/usr/src/app/vendor/cache/ruby/3.2.0/gems/que-2.3.0/lib/que/connection.rb:63:in `exec': PQsocket() can't get socket descriptor (PG::ConnectionBad)
As I see the locker file code doesn't have the re-start logic on errors (and also there is no way to put such logs to the default logger, cause it's another locker-process).
So the k8s pod always restarts...
Any thoughts or suggestions?
Much appreciated in advance!
P.S. Also I'm using the que-scheduler gem & have one job in schedule for test purposes & have Sequel as ORM.
It seems to me there is a pattern - when the Queue::Scheduler::Scheduler Job by run_at starts polling, then restarts stop... Maybe there is an issue in que-scheduler?
Interesting logs! It's strange the way the timezone changes format. Were new boxes rolling in with a new Ruby or gems?
How complex is the schedule yaml? The scheduler is nothing more than an inline single thread algorithm in a job. It spawns no new connections, processes, threads or similar. It doesn't monkey patch anything. What timezone are you in?
If you're thinking it is a problem with the scheduler do raise an issue there. But on balance I'd think it is revealing a deploy issue, memory / cpu constraints, or maybe a que issue that could happen to any other job. But I could be wrong.
It's custom logger, actually (I've forgotten to tell about). It's just an example that Queue::Scheduler::Scheduler Job works without failures.