can't access workflow logs from AoU RWB
ruchim opened this issue · 4 comments
I'm trying to use Cromshell from the AoU RWB
I ran a workflow and status succeeded (bold)
Then, when I go to get logs, it fails and the stacktrace (see below) isn't very clear to me. Trying to debug how to fix the logs functionality
> jupyter@5cbce7741997:~$ cromshell-alpha status c47be75c-ba47-4b9e-99a8-8e5cf5d70bc4
2023-05-12 16:23:52,910 cromshell.main INFO Invoked by: /opt/conda/bin/cromshell-alpha status c47be75c-ba47-4b9e-99a8-8e5cf5d70bc4
2023-05-12 16:23:52,910 cromshell.main INFO Log level set to: INFO
2023-05-12 16:23:52,910 cromshell.utilities.cromshellconfig INFO Workflow id and cromwell server not specified.Using default cromwell server
2023-05-12 16:23:52,910 cromshell.utilities.cromshellconfig INFO Server: https://notebooks.firecloud.org/proxy/google/v1/apps/terra-vpc-sc-73ac72b9/all-of-us-9852-cromwell-2473/cromwell-service
2023-05-12 16:23:52,911 cromshell.utilities.cromshellconfig INFO Setting requests timeout from value in config file.
2023-05-12 16:23:52,911 cromshell.utilities.cromshellconfig INFO Request Timeout value: 5 sec
2023-05-12 16:23:52,911 cromshell.utilities.cromshellconfig INFO Will send auth header with token for pet-26799372794334fc31363@terra-vpc-sc-73ac72b9.iam.gserviceaccount.com from config.
2023-05-12 16:23:52,911 cromshell.utilities.cromshellconfig INFO Will send referer header https://notebooks.firecloud.org from config.
2023-05-12 16:23:52,912 cromshell.status.command INFO status
2023-05-12 16:23:52,912 cromshell.utilities.cromshellconfig INFO Checking submission file for associated cromwell server with the provided workflow id.
2023-05-12 16:23:52,912 cromshell.utilities.cromshellconfig INFO Cromwell server set to matching workflow id insubmission file.
2023-05-12 16:23:52,912 cromshell.utilities.cromshellconfig INFO WorkflowID: c47be75c-ba47-4b9e-99a8-8e5cf5d70bc4
2023-05-12 16:23:52,912 cromshell.utilities.cromshellconfig INFO Server: https://notebooks.firecloud.org/proxy/google/v1/apps/terra-vpc-sc-73ac72b9/all-of-us-9852-cromwell-2473/cromwell-service__ .,-;-;-,. /'_\ _/_/_/_|_\_\) /'-<><><><>=/
`//====//-'_
"" "" ""
**> {"status":"Succeeded",
"id":"c47be75c-ba47-4b9e-99a8-8e5cf5d70bc4"}**
jupyter@5cbce7741997:~$ cromshell-alpha logs c47be75c-ba47-4b9e-99a8-8e5cf5d70bc4
2023-05-12 16:24:05,081 cromshell.main INFO Invoked by: /opt/conda/bin/cromshell-alpha logs c47be75c-ba47-4b9e-99a8-8e5cf5d70bc4
2023-05-12 16:24:05,082 cromshell.main INFO Log level set to: INFO
2023-05-12 16:24:05,082 cromshell.utilities.cromshellconfig INFO Workflow id and cromwell server not specified.Using default cromwell server
2023-05-12 16:24:05,082 cromshell.utilities.cromshellconfig INFO Server: https://notebooks.firecloud.org/proxy/google/v1/apps/terra-vpc-sc-73ac72b9/all-of-us-9852-cromwell-2473/cromwell-service
2023-05-12 16:24:05,082 cromshell.utilities.cromshellconfig INFO Setting requests timeout from value in config file.
2023-05-12 16:24:05,082 cromshell.utilities.cromshellconfig INFO Request Timeout value: 5 sec
2023-05-12 16:24:05,083 cromshell.utilities.cromshellconfig INFO Will send auth header with token for pet-26799372794334fc31363@terra-vpc-sc-73ac72b9.iam.gserviceaccount.com from config.
2023-05-12 16:24:05,083 cromshell.utilities.cromshellconfig INFO Will send referer header https://notebooks.firecloud.org from config.
2023-05-12 16:24:05,084 cromshell.logs.command INFO logs
2023-05-12 16:24:05,084 cromshell.utilities.cromshellconfig INFO Checking submission file for associated cromwell server with the provided workflow id.
2023-05-12 16:24:05,084 cromshell.utilities.cromshellconfig INFO Cromwell server set to matching workflow id insubmission file.
2023-05-12 16:24:05,084 cromshell.utilities.cromshellconfig INFO WorkflowID: c47be75c-ba47-4b9e-99a8-8e5cf5d70bc4
2023-05-12 16:24:05,084 cromshell.utilities.cromshellconfig INFO Server: https://notebooks.firecloud.org/proxy/google/v1/apps/terra-vpc-sc-73ac72b9/all-of-us-9852-cromwell-2473/cromwell-service
2023-05-12 16:24:06,164 cromshell.logs.command INFO Status keys set to ['Failed']
Traceback (most recent call last):
File "/opt/conda/bin/cromshell-alpha", line 8, in
sys.exit(main_entry())
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/click/decorators.py", line 38, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/cromshell/logs/command.py", line 75, in main
print_logs=print_logs,
File "/opt/conda/lib/python3.7/site-packages/cromshell/logs/command.py", line 102, in obtain_and_print_logs
verify_certs=config.requests_verify_certs,
TypeError: get_workflow_metadata() missing 1 required positional argument: 'headers'
Hi @ruchim,
I see you are running cromshell-alpha, could you try installing the latest version of cromshell on RW. One way to do this is through 'pip install git+https://github.com/broadinstitute/cromshell.git` and try logs again. (the recent release is called via cromshell as opposed to cromshell-alpha)
Tried an update (i can confirm i'm using cromshell 2.0.0)! The logging level is so much nicer (yay!) Still same issue with using the logs command tho
jupyter@c1b74825647d:~$ cromshell status test-1
__
.,-;-;-,. /'_\
_/_/_/_|_\_\) /
'-<_><_><_><_>=/\
`/_/====/_/-'\_\
"" "" ""
{"status":"Succeeded",
"id":"3f1287e8-6442-43e5-b8b9-a81daf4a1a33"}
jupyter@c1b74825647d:~$ cromshell logs test-1
No logs with status ['Failed'] found for workflow, try adding the argument '-s ALL' to list logs with any status
jupyter@c1b74825647d:~$ cromshell version
cromshell 2.0.0
This looks good.
By default, the logs command will retrieve the logs of tasks with the "Failed " status. Since the workflow being checked here completed with the status of "Succeeded", the logs command wont find any failed tasks and will exit with
No logs with status ['Failed'] found for workflow, try adding the argument '-s ALL' to list logs with any status
This default was set because we assume users are mainly looking at the logs of a workflow because they want to investigate a failure.