Software: Solve racing issue encountered by Celery.
Closed this issue · 1 comments
mlej8 commented
When celery executes tasks, some of the tasks are failing when writing to the database.
Traceback (most recent call last):
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1750, in _execute_context
self.dialect.do_executemany(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 962, in do_executemany
context._psycopg2_fetched_rows = xtras.execute_values(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/psycopg2/extras.py", line 1270, in execute_values
cur.execute(b''.join(parts))
psycopg2.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/celery/app/trace.py", line 450, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/dcp/utils/celery.py", line 6, in __call__
return self.run(*args, **kwargs)
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/dcp/tasks.py", line 44, in store_stream_data
db.session.commit()
File "<string>", line 2, in commit
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1428, in commit
self._transaction.commit(_to_root=self.future)
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829, in commit
self._prepare_impl()
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
self.session.flush()
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3298, in flush
self._flush(objects)
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3438, in _flush
transaction.rollback(_capture_exception=True)
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3398, in _flush
flush_context.execute()
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
util.preloaded.orm_persistence.save_obj(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj
_emit_insert_statements(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1153, in _emit_insert_statements
c = connection._execute_20(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1451, in _execute_clauseelement
ret = self._execute_context(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
self._handle_dbapi_exception(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1994, in _handle_dbapi_exception
util.raise_(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1750, in _execute_context
self.dialect.do_executemany(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 962, in do_executemany
context._psycopg2_fetched_rows = xtras.execute_values(
File "/home/mlej8/projects/NeuroTechX-McGill-2021/software/data_collection_platform/backend/env/lib/python3.8/site-packages/psycopg2/extras.py", line 1270, in execute_values
cur.execute(b''.join(parts))
sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) error with status PGRES_TUPLES_OK and no message from the libpq
Very likely due to the fact that the database connection is traveling accross processes.
https://stackoverflow.com/questions/30241911/psycopg2-error-databaseerror-error-with-no-message-from-the-libpq
mlej8 commented
@archarbar please look into this issue.