Stuck at FreeSurfer Stage (mris_fix_topolo)

Question

Stuck at FreeSurfer Stage (mris_fix_topolo)

Closed this issue 2 months ago · 3 comments

What happened?

Dear abcd-hcp-pipeline Developers,

I am currently working on processing Human Brain Network (HBN) subjects and have encountered an issue with the abcd-hcp-pipeline version 0.1.3, specifically, I am stuck at the FreeSurfer stage. Specifically, the process seems to hang indefinitely during the execution of the "mris_fix_topolo" command. This issue is observed through the top command, which shows continuous execution without progression.

I am tasked with processing over 400 subjects and have successfully processed 286 of them before encountering this problem.

What command did you use?

cat HBN_BIDS_list_1108.txt | parallel --jobs 26 docker run --rm -v /home/panshi/ABCD_Results/panshi/HBN/ABCD_HCP_pipe/ABCD-BIDS/CDODD_BIDS_raw:/bids_input:ro -v /home/panshi/Documents/ABCD-BIDS/HBN_deri_temp/:/output -v /home/panshi/Downloads/freesurfer_license/:/license dcanumn/abcd-hcp-pipeline /bids_input /output --freesurfer-license=/license/license.txt --ncpus 3 --participant-label {} >> parallel_docker_HBN_1108_newserver.log 2>&1

What version of the abcd-hcp pipeline are you using?

abcd-hcp-pipeline v0.1.3

Directory Structure

No response

Relevant log output

abcd-hcp-pipeline v0.1.3
running FreeSurfer
/opt/pipeline/FreeSurfer/FreeSurferPipeline.sh \
    --subject=NDARRR570MC0 \
    --subjectDIR=/output/sub-NDARRR570MC0/ses-None/files/T1w \
    --t1=/output/sub-NDARRR570MC0/ses-None/files/T1w/T1w_acpc_dc_restore.nii.gz \
    --t1brain=/output/sub-NDARRR570MC0/ses-None/files/T1w/T1w_acpc_dc_restore_brain.nii.gz \
    --t2=/output/sub-NDARRR570MC0/ses-None/files/T1w/T2w_acpc_dc_restore.nii.gz \
    --useT2=true \
    --printcom=
missing expected outputs from FreeSurfer
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/T1w_hires.nii.gz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/lh.EC_average
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/rh.EC_average
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/lh.aparc.a2009s.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/lh.aparc.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/rh.aparc.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/T1w_hires.masked.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aparc+aseg.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aparc.a2009s+aseg.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aseg.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aseg.hires.nii.gz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/brain.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/lh.ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/rh.ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/wm.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/wmparc.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/aseg.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.aparc.a2009s.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.aparc.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.curv.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.aparc.a2009s.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.aparc.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.curv.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.mid
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sphere
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sphere.reg
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sulc
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.thickness
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.volume
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white.deformed
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area.mid
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.curv
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.curv.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.inflated
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.orig
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sphere
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sphere.reg
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sulc
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.thickness
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.volume
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.white
Traceback (most recent call last):
  File "/app/run.py", line 397, in <module>
    _cli()
  File "/app/run.py", line 69, in _cli
    return interface(**kwargs)
  File "/app/run.py", line 393, in interface
    stage.run(ncpus)
  File "/app/pipelines.py", line 592, in run
    self.teardown(result)
  File "/app/pipelines.py", line 538, in teardown
    self.__class__.__name__)
Exception: error caught during stage: FreeSurfer

        The pipeline must choose distortion correction method based on the
        type(s) of field maps available. The type of fieldmaps you have are
        either not able to be used in the abcd-hcp-pipeline or they are
        not properly identified in the BIDS format. The pipeline does not
        account for 'phasediff', 'magnitude', and 'fieldmap' field maps
        filetypes yet. If you have 'phasediff' and 'magnitude' field maps,
        please provide the original 'phase1', 'phase2', 'magnitude1', and
        'magnitude2' field maps used to calculate those files. The pipeline
        does the calculation itself.

/home/panshi/Documents/ABCD-BIDS/HBN_deri_temp/sub-NDARBE096YK6/ses-None/logs/FreeSurfer/status.json
{
    "num_runs": 1,
    "node_status": 3,
    "comment": "stage terminated, some required files were not created."
}

Add any additional information or context about the problem here.

No response

Answer 1 · 2023-11-21T01:05:25.000Z

Hi, @fullcream9. How many of the 400 subjects have this error? It looks like this could be an error with the field maps and/or field map jsons. Does this subject have field maps and if so, do the field map jsons contain intended for fields that match the field maps with the appropriate functional run files? Are the fmap or func files/jsons different somehow from the subjects you've been able to process successfully?

Answer 2 · 2023-12-15T14:56:00.000Z

Hi,

Thank you for your response and suggestions. I've carefully checked the fmap and func files/jsons for the failed subjects and found no difference between successful subjects.

So far, I have processed 400 subjects in sequence and successfully completed 280 of them before encountering the error. So, about 120 subjects have this error.

To pinpoint the problem, I attempted to process one subject, sub-NDARRR570MC0, individually using the following command:

docker run --rm -v /home/panshi/ABCD_Results/panshi/HBN/ABCD_HCP_pipe/ABCD-BIDS/CDODD_BIDS_raw:/bids_input:ro -v /home/panshi/Documents/ABCD-BIDS/HBN_deri_temp/:/output -v /home/panshi/Downloads/freesurfer_license/:/license dcanumn/abcd-hcp-pipeline /bids_input /output --freesurfer-license=/license/license.txt --ncpus 16 --participant-label NDARRR570MC0 >> NDARRR570MC0_test_newserver.log 2>&1

It appears that the issue arises during the FreeSurfer recon-all step (mris_fix_topolo, in particular). I'm attaching the NDARRR570MC0_test_newserver.log and recon-all.err files for your reference and further insight.

Any additional guidance or insights you could provide would be greatly appreciated.

NDARRR570MC0_test_newserver.log:

    The pipeline must choose distortion correction method based on the
    type(s) of field maps available. The type of fieldmaps you have are
    either not able to be used in the abcd-hcp-pipeline or they are
    not properly identified in the BIDS format. The pipeline does not
    account for 'phasediff', 'magnitude', and 'fieldmap' field maps
    filetypes yet. If you have 'phasediff' and 'magnitude' field maps,
    please provide the original 'phase1', 'phase2', 'magnitude1', and
    'magnitude2' field maps used to calculate those files. The pipeline
    does the calculation itself.

abcd-hcp-pipeline v0.1.3
running PreFreeSurfer
/opt/pipeline/PreFreeSurfer/PreFreeSurferPipeline.sh
--path=/output/sub-NDARRR570MC0/ses-None/files
--subject=NDARRR570MC0
--t1=/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-HCP_T1w.nii.gz@/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNav_T1w.nii.gz@/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNavNorm_T1w.nii.gz
--t2=/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNav_T2w.nii.gz@/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNavNorm_T2w.nii.gz
--t1template=/opt/pipeline/global/templates/MNI152_T1_1mm.nii.gz
--t1templatebrain=/opt/pipeline/global/templates/MNI152_T1_1mm_brain.nii.gz
--t1template2mm=/opt/pipeline/global/templates/MNI152_T1_2mm.nii.gz
--t2template=/opt/pipeline/global/templates/MNI152_T2_1mm.nii.gz
--t2templatebrain=/opt/pipeline/global/templates/MNI152_T2_1mm_brain.nii.gz
--t2template2mm=/opt/pipeline/global/templates/MNI152_T2_2mm.nii.gz
--templatemask=/opt/pipeline/global/templates/MNI152_T1_1mm_brain_mask.nii.gz
--template2mmmask=/opt/pipeline/global/templates/MNI152_T1_2mm_brain_mask_dil.nii.gz
--brainsize=150
--fnirtconfig=/opt/pipeline/global/config/T1_2_MNI152_2mm.cnf
--fmapmag=NONE
--fmapphase=NONE
--fmapgeneralelectric=NONE
--echodiff=NONE
--SEPhaseNeg=NONE
--SEPhasePos=NONE
--echospacing=NONE
--seunwarpdir=NONE
--t1samplespacing=0.000012019
--t2samplespacing=0.000008138
--unwarpdir=z
--gdcoeffs=NONE
--avgrdcmethod=NONE
--topupconfig=/opt/pipeline/global/config/b02b0.cnf
--useT2=true
--printcom=
--useStudyTemplate=false
--StudyTemplate=NONE
--StudyTemplateBrain=NONE
abcd-hcp-pipeline v0.1.3
running FreeSurfer
/opt/pipeline/FreeSurfer/FreeSurferPipeline.sh
--subject=NDARRR570MC0
--subjectDIR=/output/sub-NDARRR570MC0/ses-None/files/T1w
--t1=/output/sub-NDARRR570MC0/ses-None/files/T1w/T1w_acpc_dc_restore.nii.gz
--t1brain=/output/sub-NDARRR570MC0/ses-None/files/T1w/T1w_acpc_dc_restore_brain.nii.gz
--t2=/output/sub-NDARRR570MC0/ses-None/files/T1w/T2w_acpc_dc_restore.nii.gz
--useT2=true
--printcom=
missing expected outputs from FreeSurfer
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/T1w_hires.nii.gz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/lh.EC_average
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/rh.EC_average
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/lh.aparc.a2009s.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/lh.aparc.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/rh.aparc.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/T1w_hires.masked.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aparc+aseg.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aparc.a2009s+aseg.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aseg.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aseg.hires.nii.gz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/brain.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/lh.ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/rh.ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/wm.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/wmparc.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/aseg.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.aparc.a2009s.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.aparc.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.curv.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.aparc.a2009s.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.aparc.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.curv.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.mid
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sphere
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sphere.reg
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sulc
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.thickness
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.volume
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white.deformed
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area.mid
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.curv
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.curv.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.inflated
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.orig
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sphere
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sphere.reg
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sulc
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.thickness
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.volume
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.white
Traceback (most recent call last):
File "/app/run.py", line 397, in
_cli()
File "/app/run.py", line 69, in _cli
return interface(**kwargs)
File "/app/run.py", line 393, in interface
stage.run(ncpus)
File "/app/pipelines.py", line 592, in run
self.teardown(result)
File "/app/pipelines.py", line 538, in teardown
self.class.name)
Exception: error caught during stage: FreeSurfer

recon-all.err：

SUBJECT NDARRR570MC0
DATE Fri Dec 15 03:11:12 UTC 2023
USER root
HOST 569e2b72c878
PROCESSOR x86_64
OS Linux
Linux 569e2b72c878 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 2 18:01:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$Id: recon-all,v 1.379.2.73 2013/05/12 23:15:37 nicks Exp $
/opt/freesurfer/bin/recon-all
PWD /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/scripts
CMD mris_fix_topology -mgz -sphere qsphere.nofix -ga -seed 1234 NDARRR570MC0 lh

Answer 3 · 2024-09-18T01:45:14.000Z

@fullcream9 I'm so sorry this fell off my radar, I somehow never received a notification that you had responded. Are you still working on this? Off the top of my head:

choose only one of the T1 and T2 files to process with. it looks like you are using all of the inputs, which means you are averaging normalized and not-normalized images and also perhaps images with different acquisition protocols. for example, for T1, only use /bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNavNorm_T1w.nii.gz
are there any differences in the input data between the subjects that pass and subjects that fail? can you show an example directory structure and file contents from a subject/session for each?
try using most recent version of pipeline
is there any difference in the quality of the raw data between successful and failed subjects? did the raw data undergo some type of quality assessment before processing?

I'm going to close this for now since it's been so long, but please open it again if you would like more help/have updates! Apologies again!