Unhandled Exception when swift storage path does not exist
Closed this issue · 0 comments
jennydaman commented
Lines 1458 to 1461 in cf3c920
Try the pfurl command from the wiki page
https://github.com/FNNDSC/pfcon/wiki/pfcon-FS-and-DS-plugin-example#call
but without having pushed anything to swift storage first.
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.do_POST() |
<----
Host: 192.168.1.240:5005
User-Agent: PycURL/7.43.0.5 libcurl/7.70.0 OpenSSL/1.1.1g zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.40.0
Accept: */*
Mode: control
Content-Length: 1101
Content-Type: application/x-www-form-urlencoded
<----
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.do_POST() | data length = 1101
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.do_POST() | data: b'{"payload": {"action": "coordinate", "meta-compute": {"auid": "chris", "cmd": "python3 /usr/src/dircopy/dircopy.py --saveinputmeta --saveoutputmeta --dir /share/incoming /share/outgoing", "container": {"manager": {"app": "swarm.py", "env": {"meta-store": "key", "serviceName": "1", "serviceType": "docker", "shareDir": "%shareDir"}, "image": "fnndsc/swarm"}, "target": {"cmdParse": false, "execshell": "python3", "image": "fnndsc/pl-dircopy", "selfexec": "dircopy.py", "selfpath": "/usr/src/dircopy"}}, "cpu_limit": "1000m", "gpu_limit": 0, "jid": "1", "memory_limit": "200Mi", "number_of_workers": "1", "service": "host", "threaded": true}, "meta-data": {"localSource": {"path": "chris/uploads/DICOM/dataset1", "storageType": "swift"}, "localTarget": {"createDir": true, "path": "chris/feed_1/dircopy_1/data"}, "remote": {"key": "%meta-store"}, "service": "host", "specialHandling": {"cleanup": true, "op": "plugin"}, "transport": {"compress": {"archive": "zip", "cleanup": true, "unpack": true}, "mechanism": "compress"}}, "meta-store": {"key": "jid", "meta": "meta-compute"}, "threadAction": true}}'
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.do_POST() | form length = 0
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.do_POST() | Parsing JSON data...
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.do_POST() | d_msg = { 'action': 'coordinate',
'meta-compute': { 'auid': 'chris',
'cmd': 'python3 /usr/src/dircopy/dircopy.py '
'--saveinputmeta --saveoutputmeta --dir '
'/share/incoming /share/outgoing',
'container': { 'manager': { 'app': 'swarm.py',
'env': { 'meta-store': 'key',
'serviceName': '1',
'serviceType': 'docker',
'shareDir': '%shareDir'},
'image': 'fnndsc/swarm'},
'target': { 'cmdParse': False,
'execshell': 'python3',
'image': 'fnndsc/pl-dircopy',
'selfexec': 'dircopy.py',
'selfpath': '/usr/src/dircopy'}},
'cpu_limit': '1000m',
'gpu_limit': 0,
'jid': '1',
'memory_limit': '200Mi',
'number_of_workers': '1',
'service': 'host',
'threaded': True},
'meta-data': { 'localSource': { 'path': 'chris/uploads/DICOM/dataset1',
'storageType': 'swift'},
'localTarget': { 'createDir': True,
'path': 'chris/feed_1/dircopy_1/data'},
'remote': {'key': '%meta-store'},
'service': 'host',
'specialHandling': {'cleanup': True, 'op': 'plugin'},
'transport': { 'compress': { 'archive': 'zip',
'cleanup': True,
'unpack': True},
'mechanism': 'compress'}},
'meta-store': {'key': 'jid', 'meta': 'meta-compute'},
'threadAction': True}
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.do_POST() | verb: coordinate detected.
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.do_POST() | method to call: coordinate_process(request = d_msg)
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.coordinate_process() | coordinate_process()
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.key_dereference() | key_dereference()
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.key_dereference() | key = 1
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.pushData_handler() | metaData = { 'local': {'path': 'chris/uploads/DICOM/dataset1', 'storageType': 'swift'},
'localSource': { 'path': 'chris/uploads/DICOM/dataset1',
'storageType': 'swift'},
'localTarget': {'createDir': True, 'path': 'chris/feed_1/dircopy_1/data'},
'remote': {'key': '1'},
'service': 'host',
'specialHandling': {'cleanup': True, 'op': 'plugin'},
'transport': { 'compress': { 'archive': 'zip',
'cleanup': True,
'unpack': True},
'mechanism': 'compress'}}
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'not started'},
'swiftPut': {'return': {}, 'status': 'not started'}}
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.dataRequest_process() | dataRequest_process()
2020-06-04 09:57:54 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'pushing'},
'swiftPut': {'return': {}, 'status': 'not started'}}
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/dist-packages/pfcon/pfcon.py", line 427, in dataRequest_process
d_pushPath = self.dataRequest_processPushPath(d_meta = d_meta)
File "/usr/local/lib/python3.6/dist-packages/pfcon/pfcon.py", line 383, in dataRequest_processPushPath
fromLocation = d_local['path']
File "/usr/local/lib/python3.6/dist-packages/pfcon/pfcon.py", line 1634, in swiftstorage_objPull
d_lsSwift = self.swiftstorage_ls(path = str_swiftLocation)
File "/usr/local/lib/python3.6/dist-packages/pfcon/pfcon.py", line 1461, in swiftstorage_ls
full_listing = True)[1]
File "/usr/local/lib/python3.6/dist-packages/swiftclient/client.py", line 1820, in get_container
query_string=query_string)
File "/usr/local/lib/python3.6/dist-packages/swiftclient/client.py", line 1739, in _retry
service_token=self.service_token, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/swiftclient/client.py", line 946, in get_container
service_token=service_token, headers=headers)
File "/usr/local/lib/python3.6/dist-packages/swiftclient/client.py", line 993, in get_container
raise ClientException.from_response(resp, 'Container GET failed', body)
swiftclient.exceptions.ClientException: Container GET failed: http://swift_service:8080/v1/AUTH_chris/users?format=json&prefix=chris/uploads/DICOM/dataset1 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<'
2020-06-04 09:57:55 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'pushing'},
'swiftPut': {'return': {}, 'status': 'not started'}}
2020-06-04 09:57:56 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'pushing'},
'swiftPut': {'return': {}, 'status': 'not started'}}
2020-06-04 09:57:57 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'pushing'},
'swiftPut': {'return': {}, 'status': 'not started'}}
2020-06-04 09:57:58 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'pushing'},
'swiftPut': {'return': {}, 'status': 'not started'}}
2020-06-04 09:57:59 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'pushing'},
'swiftPut': {'return': {}, 'status': 'not started'}}
2020-06-04 09:58:00 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'pushing'},
'swiftPut': {'return': {}, 'status': 'not started'}}
2020-06-04 09:58:01 | 48c8eafa7004 | pfcon.py:StoreHandler.jobStatus_do() | d_info = { 'compute': {'return': {}, 'status': 'not started'},
'pullPath': {'return': {}, 'status': 'not started'},
'pushPath': {'return': {}, 'status': 'pushing'},
'swiftPut': {'return': {}, 'status': 'not started'}}
infinite loop of pfcon.py:StoreHandler.jobStatus_do()...