MySQL Connection is going away from Delayed Job and Delayed job is not restarting automatically
rajaselvam26 opened this issue · 5 comments
All,
In our production server we noticed that delayed job is getting stopped after certain time. When we analyzed the issue We noticed the following error in our server: Error while reserving job: Mysql2::Error::ConnectionError: MySQL server has gone away:
We noticed the above issue repeatedly in all of our servers. Do we have any solution or workaround for this issue?
Our Environment:
- rails 5.2.3
- delayed_job_active_record 4.1.3
- Ruby 2.5.5
Please see the logs below:
I, [2019-11-01T05:36:02.610595 #228] INFO -- : 2019-11-01T05:36:02+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: Mysql2::Error::ConnectionError: MySQL server has gone away: UPDATE delayed_jobs SET delayed_jobs.locked_at = '2019-11-01 05:36:02', delayed_jobs.locked_by = 'delayed_job host:f5b8fa0615e6 pid:228' WHERE ((run_at <= '2019-11-01 05:36:02.608564' AND (locked_at IS NULL OR locked_at < '2019-11-01 01:36:02.608583') OR locked_by = 'delayed_job host:f5b8fa0615e6 pid:228') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1
I, [2019-11-01T05:36:07.622080 #228] INFO -- : 2019-11-01T05:36:07+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
I, [2019-11-01T05:36:12.631055 #228] INFO -- : 2019-11-01T05:36:12+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
I, [2019-11-01T05:36:17.638905 #228] INFO -- : 2019-11-01T05:36:17+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
I, [2019-11-01T05:36:22.649648 #228] INFO -- : 2019-11-01T05:36:22+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
I, [2019-11-01T05:36:27.658382 #228] INFO -- : 2019-11-01T05:36:27+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
I, [2019-11-01T05:36:32.666572 #228] INFO -- : 2019-11-01T05:36:32+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
I, [2019-11-01T05:36:37.672296 #228] INFO -- : 2019-11-01T05:36:37+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
I, [2019-11-01T05:36:42.683623 #228] INFO -- : 2019-11-01T05:36:42+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
I, [2019-11-01T05:36:47.693297 #228] INFO -- : 2019-11-01T05:36:47+0000: [Worker(delayed_job host:f5b8fa0615e6 pid:228)] Error while reserving job: MySQL client is not connected
F, [2019-11-01T05:36:47.694147 #228] FATAL -- : MySQL client is not connected
I, [2019-11-01T05:39:41.214553 #97] INFO -- : [5fedea19-72b2-4b67-a0ec-09d34a5f1504] Started GET "/" for 172.16.2.1 at 2019-11-01 05:39:41
Thanks,
Rajaselvam
The most common issue is restarting mysql which kills all the connections or an unstable network connection to your database server. ActiveRecord does a relatively poor job of recovering from these connection issues.
Hi David (@albus522 ,
We are not disturbing the MySQL server as its running in another instance. When I check the status of MySQL its showing as running only and I couldn't see any evidence of restart over there, I believe its nothing related to ActiveRecorsd. I guess it could be of Delayed job daemons.
Thanks,
Rajaselvam. M
If you search MySQL server has gone away you will find it is a very common issue when using mysql. There are loads of reasons it can be triggered but I can say it isn't something DJ or daemons is specifically doing or not doing.
Also, DJ does not have any automatic restart feature and never has. You need to use an external process monitor such as monit, god, or one of the many others to detect the worker process has died and restart it.
Thanks for your reply albus522.., Will try to use external process for monitoring DJ