compneuro-ncu/fmridenoise

fmridenoise.interfaces.bids.MissingFile: missing file(s)

costantinoai opened this issue · 11 comments

costantinoai commented

When trying to run from docker the following errors comes up:

PS C:\Users\45027900> docker run -it --rm -v D:\Andrea\fov\BIDS:/BIDS -v D:\Andrea\fov\BIDS\derivatives\fmriprep:/derivatives fmridenoise compare /BIDS -d /derivatives --MultiProc -t exp
/usr/local/lib/python3.8/dist-packages/numpy-1.16.4-py3.8-linux-x86_64.egg/numpy/ma/core.py:4466: SyntaxWarning: "is" with a literal. Did you mean "=="?
if self.shape is ():
201219-12:14:44,624 nipype.workflow INFO:
[Node] Setting-up "BidsValidate" in "/tmp/tmp287wv_of/BidsValidate".
201219-12:14:44,627 nipype.workflow INFO:
[Node] Running "BidsValidate" ("fmridenoise.interfaces.bids.BIDSValidate")
/usr/local/lib/python3.8/dist-packages/pybids-0.12.4-py3.8.egg/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the config_filename, ignore, force_index, and index_metadata arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the indexer argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
201219-12:14:55,358 nipype.workflow WARNING:
Storing result file without outputs
201219-12:14:55,358 nipype.workflow WARNING:
[Node] Error on "BidsValidate" (/tmp/tmp287wv_of/BidsValidate)
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/fmridenoise-src/fmridenoise/main.py", line 236, in
main()
File "/fmridenoise-src/fmridenoise/main.py", line 228, in main
compare(args)
File "/fmridenoise-src/fmridenoise/main.py", line 177, in compare
workflow = init_fmridenoise_wf(input_dir,
File "/fmridenoise-src/fmridenoise/workflows/base.py", line 455, in init_fmridenoise_wf
result = bids_validate.run()
File "/usr/local/lib/python3.8/dist-packages/nipype-1.6.0-py3.8.egg/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/lib/python3.8/dist-packages/nipype-1.6.0-py3.8.egg/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/lib/python3.8/dist-packages/nipype-1.6.0-py3.8.egg/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/lib/python3.8/dist-packages/nipype-1.6.0-py3.8.egg/nipype/interfaces/base/core.py", line 434, in run
runtime = self._run_interface(runtime)
File "/fmridenoise-src/fmridenoise/interfaces/bids.py", line 405, in _run_interface
entities_files, (tasks, sessions, subjects, runs) = BIDSValidate.validate_files(
File "/fmridenoise-src/fmridenoise/interfaces/bids.py", line 359, in validate_files
raise MissingFile(
fmridenoise.interfaces.bids.MissingFile: missing file(s) for {'subject': '05', 'task': 'exp', 'run': 5, 'extension': 'tsv', 'suffix': 'timeseries', 'desc': 'confounds'} (check if you are using AROMA pipelines)

this subject didn't have any run 5, and this is tru for other subjects. how can i solve this?

the same happens if i remove -t exp : i have another task ('loc') with only one run, but the bids validator is looking for 5 runs (it's wrongly assuming that the number of runs would be the same for loc and exp, maybe?)

SiegfriedWagner commented

It's 'wrongly' (I will call it 'as creators intended') assuming that if you provide e.g --run 1 2 3 4 5 as input it will look for data in all combinations of five runs, all task, all sessions and all subjects provided at input.

You may consider this as a flaw because it makes our tool not flexible enough.

There is idea for future release to make fmridenoise work with 'incomplete' (to some degree) datasets but this requires major rewiring of dataflow (basically splitting that into 4 separate workflows run one after other based on data aggregation joins).

I cannot estimate when flexibility will be introduced... we got limited time to develop fmridenoise.

costantinoai commented
SiegfriedWagner commented

I was thinking about it while you responded - there are no comparisons between task, sessions and runs (only between pipelines on groups of subjects), so you can run fmridenoise separately for each task, session and run combination (or any valid subset of those). Only disadvantage will be lots of separate reports instead of one.

costantinoai commented
costantinoai commented

So, I tried to run fmridenoise on the loc task only, but the tool is still looking for loc run 2 (that does not exist):

C:\Users\45027900> docker run -it --rm -v D:\Andrea\fov\BIDS:/BIDS -v D:\Andrea\fov\BIDS\derivatives\fmriprep:/derivatives fmridenoise compare /BIDS -d /derivatives --MultiProc -t loc
/usr/local/lib/python3.8/dist-packages/numpy-1.16.4-py3.8-linux-x86_64.egg/numpy/ma/core.py:4466: SyntaxWarning: "is" with a literal. Did you mean "=="?
if self.shape is ():
201220-00:53:35,168 nipype.workflow INFO:
[Node] Setting-up "BidsValidate" in "/tmp/tmpbbownsuf/BidsValidate".
201220-00:53:35,171 nipype.workflow INFO:
[Node] Running "BidsValidate" ("fmridenoise.interfaces.bids.BIDSValidate")
/usr/local/lib/python3.8/dist-packages/pybids-0.12.4-py3.8.egg/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the config_filename, ignore, force_index, and index_metadata arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the indexer argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
201220-00:54:06,428 nipype.workflow WARNING:
Storing result file without outputs
201220-00:54:06,429 nipype.workflow WARNING:
[Node] Error on "BidsValidate" (/tmp/tmpbbownsuf/BidsValidate)
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/fmridenoise-src/fmridenoise/main.py", line 236, in
main()
File "/fmridenoise-src/fmridenoise/main.py", line 228, in main
compare(args)
File "/fmridenoise-src/fmridenoise/main.py", line 177, in compare
workflow = init_fmridenoise_wf(input_dir,
File "/fmridenoise-src/fmridenoise/workflows/base.py", line 455, in init_fmridenoise_wf
result = bids_validate.run()
File "/usr/local/lib/python3.8/dist-packages/nipype-1.6.0-py3.8.egg/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/lib/python3.8/dist-packages/nipype-1.6.0-py3.8.egg/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/lib/python3.8/dist-packages/nipype-1.6.0-py3.8.egg/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/lib/python3.8/dist-packages/nipype-1.6.0-py3.8.egg/nipype/interfaces/base/core.py", line 434, in run
runtime = self._run_interface(runtime)
File "/fmridenoise-src/fmridenoise/interfaces/bids.py", line 405, in _run_interface
entities_files, (tasks, sessions, subjects, runs) = BIDSValidate.validate_files(
File "/fmridenoise-src/fmridenoise/interfaces/bids.py", line 359, in validate_files
raise MissingFile(
fmridenoise.interfaces.bids.MissingFile: missing file(s) for {'subject': '02', 'task': 'loc', 'run': 2, 'extension': 'tsv', 'suffix': 'timeseries', 'desc': 'confounds'} (check if you are using AROMA pipelines)

how can I tell to look just for run 1? and why is looking for run 2 if only task 'exp' has more than 1 run?

SiegfriedWagner commented
costantinoai commented

Thanks for your help. I will try and get back to you. In the meantime I got another problem, but I will open a separate issue for that.

Just a note about the bids selector and the AROMA files: I downloaded and built the new dev code on docker, but when I run fmridenoise compare it looks like (as you can see from the error above) it is looking for files with the AROMA correction, but I never specified that AROMA was applied.

My understanding is that fmridenoise should look for AROMA files only if speficied with the aroma flag in command line. In my case I didn't, so I suppose that the error comes up for something in the code that forces looking for AROMA files. I made some changes in the bids.py (I set True for the no_aroma boolean, and False for the aroma boolean) file, and these changes appear to solve the problem.

SiegfriedWagner commented
costantinoai commented

I am sorry, I am not sure I got it right.

From the usage in the docs, I read that

--use-aroma Run denoising pipelines based on ICA-AROMA, default False

are you saying that this is not up to date? if that's the case, how do I specify that I didn't use aroma (since it seems it is looking for aroma files)?

thanks

SiegfriedWagner commented
costantinoai commented

Got it. I will try!

thanks