Database lock
Opened this issue · 9 comments
If the database is very busy with other stuff while saving, i have often experienced that chroniker is unable to release lock on the database, leaving the module in a state where no cron's can be run, untill the database releases the lock due to timeout.
Is there any workaround this?
ERROR 2016-01-13 12:40:39,984 base 29710 140455342683968 Internal Server Error: /admin/chroniker/job/25/stop/
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", li ne 114, in get_response
response = wrapped_callback(request, _callback_args, *_callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/utils/decorators.py", line 99, in _wrapped_view
response = view_func(request, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/cache.py" , line 52, in _wrapped_view_func
response = view_func(request, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/admin/sites.py", l ine 198, in inner
return view(request, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/chroniker/admin.py", line 350, in stop_job_view
force_stop = True,
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 490, in update
rows = query.get_compiler(self.db).execute_sql(None)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py" , line 975, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py" , line 781, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 99, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py" , line 124, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction ')
after lock is placed i get these errors no matter what i do. Only reboot solves the issue
Did you try enabling select for update? You shouldn't have to reboot.
MySQL's default timeout is less than a minute.
On Jan 13, 2016 6:41 AM, "steffenmandrup" notifications@github.com wrote:
ERROR 2016-01-13 12:40:39,984 base 29710 140455342683968 Internal Server
Error: /admin/chroniker/job/25/stop/
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", li
ne 114, in get_response
response = wrapped_callback(request, _callback_args, *_callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/utils/decorators.py",
line 99, in_wrapped_view response = view_func(request, args, *kwargs) File
"/usr/local/lib/python2.7/dist-packages/django/views/decorators/cache.py" ,
line 52, in wrapped_view_func response = view_func(request, args, *kwargs)
File
"/usr/local/lib/python2.7/dist-packages/django/contrib/admin/sites.py", l
ine 198, in inner return view(request, args, *kwargs) File
"/usr/local/lib/python2.7/dist-packages/chroniker/admin.py", line 350, in
stop_job_view force_stop = True, File
"/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line
490, in update rows = query.get_compiler(self.db).execute_sql(None) File
"/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py" ,
line 975, in execute_sql cursor = super(SQLUpdateCompiler,
self).execute_sql(result_type) File
"/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py" ,
line 781, in execute_sql cursor.execute(sql, params) File
"/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line
53, in execute return self.cursor.execute(sql, params) File
"/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 99, in
_exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py",
line 53, in execute
return self.cursor.execute(sql, params)
File
"/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py" ,
line 124, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line
205, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line
36, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1205, 'Lock wait timeout exceeded; try restarting
transaction ')after lock is placed i get these errors no matter what i do. Only reboot
solves the issue—
Reply to this email directly or view it on GitHub
#49 (comment)
.
But why do these locks happen ? i often get them stopping for maybe 8-10 hours, before starting up again.
It's hard to say. Something might be crashing or hanging, causing a
transaction to not be committed. Do you see anything unusual in the error
logs? Another cause could be that there are multiple chroniker jobs are
running and trying to update their status.
Try added CHRONIKER_SELECT_FOR_UPDATE = True in your settings and see if
that helps.
On Wed, Jan 13, 2016 at 10:45 AM, steffenmandrup notifications@github.com
wrote:
But why do these locks happen ? i often get them stopping for maybe 8-10
hours, before starting up again.—
Reply to this email directly or view it on GitHub
#49 (comment)
.
Isn't the point with chroniker to have an easy way to setup all your cron jobs ? :) Hopefully they don't lock the database while saving their status, thus breaking if other cron jobs try to do the same. Running multiple cron jobs, you should eventually hit the point where they update at the same time. I got several cron's running every 10 min, with different process lengths.
Tried adding the setting you suggested, will post back if it helps :)
This bug still remains =/
The database ends up locked and all jobs freeze and will not run. Debug logs says nothing, so I have absolutely no idea why.
Everytime it happens, one of the cronjobs are stuck on "is fresh" = false and i'm unable to do anything.
Chroniker was moved to it's dedicated mysql instance, so must be chroniker locking something somewhere.
I have 16 cron jobs running in it fairly often - shouldn't it be possible to have several running at the same time ?
CHRONIKER_SELECT_FOR_UPDATE makes no difference
Even if a deadlock occurs, Chroniker shouldn't freeze. Django/MySQL will eventually throw a DatabaseError for the deadlock when that happens. However, I did notice a similar event on one of my systems, where a job was stuck on running but was still recording a heartbeat, indicating it hadn't frozen or encountered a deadlock. Unfortunately, it hasn't happened since, and I've now set a timeout on the job to kill it after an hour.
Let me know if you find a way to reliably reproduce this, and I'll write a unittest for it.
My entire database gets locked, so i can't even access the jobs in the django admin. It requires a mysql reboot to open it up again.
Tried to reproduce, but it seems totally random
The error always happens after is fresh = false, could be a pointer