Testtuple execution failed
Closed this issue · 0 comments
samlesu commented
[backend-org-1-substra-backend-server-54b7879994-j262b substra-backend] INFO - 2019-12-16 15:27:30,977 - events.apps - Processing task 3298e66712747719fd670cba24164479a4c778fa217856e441c98a6ca98ee770: type=testtuple status=todo with tx status: MVCC_READ_CONFLICT
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:30,994: INFO/MainProcess] Received task: substrapp.tasks.tasks.prepare_tuple[3298e66712747719fd670cba24164479a4c778fa217856e441c98a6ca98ee770]
[backend-org-2-substra-backend-server-5bcfbc8888-7tdqh substra-backend] INFO - 2019-12-16 15:27:31,007 - events.apps - Processing task 3298e66712747719fd670cba24164479a4c778fa217856e441c98a6ca98ee770: type=testtuple status=todo with tx status: MVCC_READ_CONFLICT
[backend-org-2-substra-backend-server-5bcfbc8888-7tdqh substra-backend] DEBUG - 2019-12-16 15:27:31,007 - events.apps - Skipping task 3298e66712747719fd670cba24164479a4c778fa217856e441c98a6ca98ee770: owner does not match (MyOrg1MSP vs MyOrg2MSP)
[backend-org-2-substra-backend-worker-64d655d689-cf7kg worker] [2019-12-16 15:27:31,010: ERROR/ForkPoolWorker-1] MVCC read conflict for ('logSuccessCompositeTrain', ['{"key": "503e6b9b0700c9f857cdf750d641e5d758720b93e9d8bbb6fb6836fdc8a3c849", "log": "", "outHeadModel": {"hash": "5cfbe4ac6f29b0a34fa881c209239fc91f547f89327deaa5a6dea5026eab0e68", "storageAddress": "http://substra-backend.node-2.com/model/5cfbe4ac6f29b0a34fa881c209239fc91f547f89327deaa5a6dea5026eab0e68/file/"}, "outTrunkModel": {"hash": "4af8d42042e5a3ad19bd12e98e1ff731c56a856a5f9263bb96cfc497c45e00c4", "storageAddress": "http://substra-backend.node-2.com/model/4af8d42042e5a3ad19bd12e98e1ff731c56a856a5f9263bb96cfc497c45e00c4/file/"}}'])
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,027: INFO/ForkPoolWorker-1] DISCOVERY: adding channel peers query
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,027: INFO/ForkPoolWorker-1] DISCOVERY: adding config query
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,027: INFO/ForkPoolWorker-1] DISCOVERY: adding chaincodes/collection query
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,103: INFO/ForkPoolWorker-1] create peer delivery stream
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,107: INFO/ForkPoolWorker-1] create peer delivery stream
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,134: INFO/MainProcess] Received task: substrapp.tasks.tasks.compute_task[28fc383e-dfd7-4368-bc46-29f0d173afc7]
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,145: INFO/ForkPoolWorker-1] Task substrapp.tasks.tasks.prepare_tuple[8dc933581e8e54c448c6a288f291d60625a0fd0ebcd7acf117d25ce4cce62d89] succeeded in 0.19399090000115393s: None
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,174: INFO/ForkPoolWorker-1] DISCOVERY: adding channel peers query
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,174: INFO/ForkPoolWorker-1] DISCOVERY: adding config query
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,174: INFO/ForkPoolWorker-1] DISCOVERY: adding chaincodes/collection query
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] [2019-12-16 15:27:31,206: ERROR/ForkPoolWorker-1] Task substrapp.tasks.tasks.prepare_tuple[3298e66712747719fd670cba24164479a4c778fa217856e441c98a6ca98ee770] raised unexpected: update testtuple 3298e66712747719fd670cba24164479a4c778fa217856e441c98a6ca98ee770 failed: cannot change status from waiting to doing
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] Traceback (most recent call last):
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 382, in trace_task
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] R = retval = fun(*args, **kwargs)
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 641, in __protected_call__
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] return self.run(*args, **kwargs)
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/src/app/substrapp/tasks/tasks.py", line 449, in prepare_tuple
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] log_start_tuple(tuple_type, subtuple['key'])
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/src/app/substrapp/ledger_utils.py", line 330, in log_start_tuple
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] _update_tuple_status(tuple_type, tuple_key, 'doing')
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/src/app/substrapp/ledger_utils.py", line 324, in _update_tuple_status
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] update_ledger(fcn=invoke_fcn, args=invoke_args, sync=True)
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/src/app/substrapp/ledger_utils.py", line 107, in _wrapper
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] return fn(*args, **kwargs)
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/src/app/substrapp/ledger_utils.py", line 233, in update_ledger
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] return _invoke_ledger(*args, **kwargs)
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/src/app/substrapp/ledger_utils.py", line 212, in _invoke_ledger
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] response = call_ledger('invoke', fcn=fcn, args=args, kwargs=params)
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] File "/usr/src/app/substrapp/ledger_utils.py", line 196, in call_ledger
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] raise exception_class.from_response(response)
[backend-org-1-substra-backend-worker-859bdb6d6-q9c6g worker] substrapp.ledger_utils.LedgerResponseError: update testtuple 3298e66712747719fd670cba24164479a4c778fa217856e441c98a6ca98ee770 failed: cannot change status from waiting to doing