Error copying DICOM file from my folders to the folders created by Ironsmith
Closed this issue · 17 comments
Hi,
I have been trying to run Ironsmith, but I am running into an issue when the program tries to copy DICOM files from my input into the output folder it creates. I figured out that Ironsmith is adding a hidden character to the subject id (which is the name of the folder it creates). Is this supposed to happen, or is something else going weird on my end?
This is what the code looks like:
Copying DICOM files from /inputfiles/subjectID/SWI_Images/ to /outputfiles/trial 1/subjectID/QSM/QSM_DICOM
cp: cannot create regular file 'QSM_DICOM/': Not a directory
The above line repeats many times, and then the following:
Converting QSM DICOMS in /DataDisk/ironsmith_data/ironsmith_output/1301L2_permissionstrial/1301L2/QSM/QSM_DICOM/ to NIFTI (.nii.gz)
Error: Unable to find any DICOM images in QSM_DICOM (or subfolders 5 deep)
Chris Rorden's dcm2niiX version v1.0.20210317 GCC6.3.0 x86-64 (64-bit Linux)
Any ideas would be helpful. Thanks!
Greetings,
Let's see if we can fix this. It looks like the issue might be occurring in MEDI.sh,
Specifically at this portion of the code:
cd $OutFolder/$Subj
mkdir QSM
cd QSM
mkdir QSM_DICOM
#mkdir MEDI_Output
#mkdir MEDI_mat_Files
echo ""
echo "Copying DICOM files from $QSM_Dicom_Dir to $OutFolder/$Subj/QSM/QSM_DICOM"
echo ""
singularity run -e --bind $QSM_Dicom_Dir:/mnt $Path/Functions/QSM_Container.simg find /mnt -mindepth 1 -maxdepth 1 -type f ! -regex '.*\(nii\|json\|txt\|nii.gz\|HEAD\|BRIK\|hdr\|img\)$' -exec cp '{}' 'QSM_DICOM/' ';'
Usually the error you are getting occurs when the "mkdir" commands fail due to insufficient permissions:
mkdir QSM
cd QSM
mkdir QSM_DICOM
The output folder location where Ironsmith is trying to copy files needs to have (0770/drwxrwx---) permissions (user and group need to be able to execute/read/write files in that location). Can you check on this first?
So we have changed the permissions, and that doesn't appear to be the issue. The output folders created (QSM, QSM_DICOM, etc.) have 775 permissions. We also tried using sudo and it didn't help. Do you have any other suggestions?
Would it be possible to provide data for one random participant so I can run Ironsmith on my end to see if I can replicate your error?
Here is my e-mail
e-mail: vzachari "AT" uky.edu
It will also help me if you could provide the first few lines of File.csv. This is the file with the DICOM locations and IDs per participant that Ironsmith needs as an input.
Thank you
Would dropbox work for you? If so, would it be the same email?
Yes, dropbox should work at the same e-mail.
Ok, we'll work on making sure it's all de-identified and then send it to you.
Greetings,
I asked people who have been using Ironsmith and other Singularity-based software from UKY. In some situations, when an admin installs Singularity, individual users may not have permission to fully use it unless they are granted those permissions. A work-around for this issue is to install the apptainer-suid for your linux distribution. If you can also provide the log files inside the LogFiles directory of one of the problematic participants, it might help indicate if what I describe above is the issue.
Hello,
I've been given the highest permissions possible, so I don't think that's the issue here. I've also tried to download the log files to send to you, but I think I'm running into a similar issue as before. I get a message that it does not exist, and I think this is because of the odd way the files are being created. I am going to email you the dropbox link, and I'll also bring up appraiser-suid to the IT person I've been working with.
Issue update and fix:
Singularity, installed on Ubuntu 20.04.4 LTS using a Windows 2016 Hyper-V virtual machine, could not access main local folders. The folder that needed access was a mounted virtual disk.
One solution to this issue was to follow the guide below:
Specifically, the mounted drive/folder was added to /etc/singularity/singularity.conf : "bind path = /mounted_folder" and "enable overlay = try" was enabled for Singularity to look for folders within the bind path.
Hello,
I encountered a same problem. I edited "singularity.conf" but nothing fixed.
Singularity 3.9, installed on Ubuntu 22.04.4 LTS, win11 wsl2
Copying DICOM files from /mnt/e/ambi_visual/mri/prep/sourcedata/sub-104/QSM_mag_ph to /mnt/e/ambi_visual/mri/prep/derivatives/ironsmith/104/QSM/QSM_DICOM
INFO: Converting SIF file to temporary sandbox...
cp: cannot create regular file 'QSM_DICOM/': Not a directory
cp: cannot create regular file 'QSM_DICOM/': Not a directory
INFO: Cleaning up image...
Converting QSM DICOMS in /mnt/e/ambi_visual/mri/prep/derivatives/ironsmith/104/QSM/QSM_DICOM/ to NIFTI (.nii.gz)
INFO: Converting SIF file to temporary sandbox...
Error: Unable to find any DICOM images in QSM_DICOM (or subfolders 5 deep)
Chris Rorden's dcm2niiX version v1.0.20210317 GCC6.3.0 x86-64 (64-bit Linux)
INFO: Cleaning up image...
To make sure we have things correctly. Under:
# BIND PATH: [STRING]
# DEFAULT: Undefined
# Define a list of files/directories that should be made available from within
# the container. The file or directory must exist within the container on
# which to attach to. you can specify a different source and destination
# path (respectively) with a colon; otherwise source and dest are the same.
#
# In native mode, these are ignored if singularity is invoked with --contain except
# for /etc/hosts and /etc/localtime. When invoked with --contain and --net,
# /etc/hosts would contain a default generated content for localhost resolution.
#
# In OCI mode these are only mounted when --no-compat is specified.
you need to add:
bind path = /etc/localtime
bind path = /etc/hosts
bind path = /mnt/e #<-mounted storage on WSL2
reboot wsl2
Then, can you run the following:
singularity run -e --bind /mnt/e/ambi_visual/mri/prep/derivatives/ironsmith/104/QSM/QSM_DICOM:/mnt Path/to/QSM_Container.simg pwd
The pwd command should return the full path e.g. "/mnt/e/ambi_visual/mri/prep/derivatives/ironsmith/104/QSM/QSM_DICOM"
If not, post here and we'll see what to do next. It might be that singularity cannot see anything under /mnt in WSL2.
Thank you for your response.
I have already modified the 'singularity.conf', but when running IronSmith, I still encounter the same error.
After executing the provided Singularity command, I received the following results.
(py310) zhao@DESKTOP:/mnt/e/ambi_visual/mri/prep/code$ singularity run -e --bind /mnt/e/ambi_visual/mri/prep/derivatives/ironsmith/104/QSM/QSM_DICOM:/mnt /home/zhao/IronSmithQSM/Functions/QSM_Container.simg pwd
INFO: Converting SIF file to temporary sandbox...
/home/zhao
INFO: Cleaning up image...
If this issue is specific to Singularity or wsl2, would it be possible for me to use other installation methods to avoid this?
The issue might be that singularity under WSL2 cannot see anything under /mnt, irrespective of whether you have it defined in singularity.conf or not. I have singularity installed under WSL2 on one of my old PCs. Let me boot it up today and see how I got around this issue.
Ubuntu 18.04 on WSL2 running singularity 3.5.2 seems to work. See attached picture:
If you run the same command as I did, does Singularity display the contents of your e drive?
Thank you for your response.
singularity mount seemed to be normal.
After reinstalling Singularity, I managed to successfully run Ironsmith once. However, upon trying to run it again after an interruption, I encountered different errors. Although the file does exist, it's indicating that it cannot be found.
Interestingly, files in the same IMA format of T1 can be read.
I have reinstalled singularity and ironsmith several times and tried ubuntu 20 and 18, but the problem still exist. Could there be an issue with my operation or the data itself?
I appreciate your patient guidance.
Greetings,
Can I have the logfile for one of the erroneous runs? The logfiles are under IronsmithQSM/LogFiles. I can see several potential issues on your last post. For example, in "DICOMS FOUND" it lists a dicom file but also the 01_MPRAGE.sh file which is not accurate. Then for "NOT FOUND in specified path!", the error indicates a participant and also a folder which should not happen. Also, what version of singularity are you running?
Thank you for your response. I can successfully run Ironsmith now. I made the following changes:
(1) After git clone IronSmithQSM, choose
git checkout main
(2) DO NOT use "-" in "Column1 = Subj". e.g. "sub001" rather than "sub-001", as the latter will cause error in Matlab.
My system info: Win11, WSL2, Ubuntu 20.04.6LTS, Singularity 4.1.2, Matlab R2019b, Ironsmith v1.6 beta
Data info: Siemens prisma, .IMA
Thank you for your contribution to this amazing toolkit!