Can oliver submit jobs with dependencies?
xzhub opened this issue · 12 comments
Is your feature request related to a problem? Please describe.
Seems oliver can not submit WDL jobs with dependencies. (multiple WDL files)
Describe the solution you'd like
oliver submit should have a new parameter, like --dependencies, so user can pointing it to the WDL zip file.
Describe alternatives you've considered
it would be better, if oliver could resolve the dependencies, then submit them all togeher automatically.
Could you provide some more information about what your issue is? Oliver is currently able to handle import statements, assuming the file exists.
I run oliver like this:
oliver submit ./run-all-samples.wdl in.json
The job failed, and the error is:
Failures:
== Failure 1 ==
| Workflow input processing failed
| ERROR: Finished parsing without consuming all tokens.
However, this command runs fine:
java -Dconfig.file=slurm_docker.conf -jar ../cromwell/cromwell-48.jar run ./run-all-samples.wdl -i in.json
I am not sure what is the issue.
Can you please share the contents of in.json? There may be a field we haven't enabled processing of yet. That error message should be updated as well, because it's not very useful in this case.
The oliver error message is actually came from Cromwell. Here is the metadata of that job:
{
"workflowProcessingEvents": [
{
"cromwellId": "cromid-a53fd85",
"description": "Finished",
"timestamp": "2020-04-15T03:18:16.561Z",
"cromwellVersion": "48"
},
{
"cromwellId": "cromid-a53fd85",
"description": "PickedUp",
"timestamp": "2020-04-15T03:18:16.557Z",
"cromwellVersion": "48"
}
],
"actualWorkflowLanguageVersion": "draft-2",
"submittedFiles": {
"workflow": "/disk/prj-ncov-20200330/vd_workflow/wdl/run-all-samples.wdl",
"root": "",
"options": "{\n\n}",
"inputs": "{\"run.amp_bed\":\"/disk/prj-ncov-20200330/vd_workflow/ref/SARS-CoV-2.bed.csv\",\"run.meta_file\":\"/disk/prj-ncov-20200330/vd_workflow/ref/reference.meta\",\"run.multi_fa\":\"/disk/prj-ncov-20200330/vd_workflow/ref/reference.fasta\",\"run.output_bam_files\":false,\"run.output_dir\":\"/disk/prj-ncov-20200330/vd_workflow/output_dir_0control_2pe_case\",\"run.reference_fa\":\"/disk/prj-ncov-20200330/vd_workflow/ref/ref.fa\",\"run.samples\":\"/disk/prj-ncov-20200330/vd_workflow/wdl/input/0_control-2_pe_case-testing.samples.json\"}",
"workflowUrl": "",
"labels": "{}"
},
"calls": {},
"outputs": {},
"actualWorkflowLanguage": "WDL",
"id": "908f0065-43e8-4204-969f-0a597c8b96ba",
"inputs": {},
"labels": {
"cromwell-workflow-id": "cromwell-908f0065-43e8-4204-969f-0a597c8b96ba"
},
"submission": "2020-04-15T03:18:07.010Z",
"status": "Failed",
"failures": [
{
"causedBy": [
{
"causedBy": [],
"message": "ERROR: Finished parsing without consuming all tokens.\n\n/disk/prj-ncov-20200330/vd_workflow/wdl/run-all-samples.wdl\n^\n "
}
],
"message": "Workflow input processing failed"
}
],
"end": "2020-04-15T03:18:16.561Z",
"start": "2020-04-15T03:18:16.558Z"
}
here is the in.json:
{
"run.output_dir":"/disk/prj-ncov-20200330/vd_workflow/output_dir_0control_2pe_case",
"run.amp_bed": "/disk/prj-ncov-20200330/vd_workflow/ref/SARS-CoV-2.bed.csv",
"run.reference_fa": "/disk/prj-ncov-20200330/vd_workflow/ref/ref.fa",
"run.multi_fa": "/disk/prj-ncov-20200330/vd_workflow/ref/reference.fasta",
"run.meta_file": "/disk/prj-ncov-20200330/vd_workflow/ref/reference.meta",
"run.samples": "/disk/prj-ncov-20200330/vd_workflow/wdl/input/0_control-2_pe_case-testing.samples.json",
"run.output_bam_files": false
}
Compare the Cromwell metadata and in.json file, seems the boolean parameter did not get processed correctly.
Hey @xzhub, we're currently going through a project release today and tomorrow. There's a possibility someone could look at this more closely before Friday, but at the latest we'll get back to by then with some details 👍
I tested with this fix. I works for single workflow, which is an improvement. However, it still failed on the whole workflow, when there are dependencies.
Like I proposed in the beginning, it would be good if oliver submit could provide another argument like --dependencies, or --imports, so that I can provide the zipped WDL files. It's even better if oliver can find those dependencies automatically and submit them all together to Cromwell server automatically.
Currently, I am importing other WDL files in my main WDL file directly from the same folder, like :
import "imported.wdl" as in
the error message is:
Failed to import workflow imported.wdl.:
Bad import imported.wdl: Failed to resolve 'imported.wdl' using resolver: 'http importer (no 'relative-to' origin)' (reason 1 of 1): Relative path
test.wdl:
import "imported.wdl" as in
task t1 {
Boolean in
command {
echo ${in}
}
}
workflow run {
Boolean in
call t1 { input: in=in }
call in.f1 { input: i=in }
}
imported.wdl:
task f1 {
Boolean i
command {
echo 'f1'
echo ${i}
}
}
test.json:
{
"run.in": false
}
command:
oliver su test.wdl test.json
error:
Failures:
== Failure 1 ==
| Workflow input processing failed
| Failed to import workflow imported.wdl.:
Bad import imported.wdl: Failed to resolve 'imported.wdl' using resolver: 'http importer (no 'relative-to' origin)' (reason 1 of 1): Relative path
Hi @xzhub, thanks for making the issue. Since we've been running workflows using URLs, we haven't tested out using local workflow files and dependencies. The error you got is definitely a bug and I will add a --dependencies option for the workflow dependencies zip file. I will have a fixed release for both those soon.
Please try upgrading to version 1.3.0 and let me know if you have any issues. If you installed it with pip, it would be python -m pip install --upgrade stjudecloud-oliver.
Thanks for the prompt update! The small testing example test.wdl and imported.wdl works fine now. However, my existing real workflow failed on command 'inspect'. (It went through oliver 'submit' and finished successfully eventually.)
$ oliver i c99b218c-9b6f-41cd-b0af-59c2c618e051 --debug
[D 200421 15:34:42 api:56] GET /api/workflows/{version}/c99b218c-9b6f-41cd-b0af-59c2c618e051/metadata
[E 200421 15:34:42 errors:28] Expected key is missing! The code needs to be updated, please contact the author!
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fd6de5cbfd0>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7fd6de60fa60>, 2129439.983124965)]']
connector: <aiohttp.connector.TCPConnector object at 0x7fd6de5cba10>
I am not sure, but seems it failed at a scatter process, where multiple tasks started running. Before that point, it could generate some outputs normally.
I'll close this ticket since the dependencies issue has been resolved successfully. The above issue is another one, I'll open a new ticket on it.