About 'Unable to determine Image IO reader for certain image file'
mincheoree opened this issue · 17 comments
Hello, I am having problem with just training the dataset with baseline nnUnet.
Fabian has kindly written the conversion of dataset python file, and I just used it to
convert kits 21 dataset. However following error occurs during preprocessing.
RuntimeError: Exception thrown in SimpleITK ImageFileReader_Execute: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:105:
sitk::ERROR: Unable to determine ImageIO reader for "/home/mchangaf/nnUNetFrame/DATASET/nnUNet_raw/nnUNet_raw_data/
Task135_KiTS2021/imagesTr/case_00038_0000.nii.gz"On certain files, Exception thrown in SimpleITK ImageFileReader occurs. Should I just ignore these errors and continue training?
Thanks for bringing this to our attention. Any ideas here, @FabianIsensee, @trofimova ?
hi @mincheoree. i assume you are running "nnUNet_plan_and_preprocess -t 135" command. if so, please provide your setup with packages versioning etc. If not, please provide the operations you trying to run so we can help you out.
hi @trofimova Thanks for your reply.
yes, I am running that command.
I am running this on Ubuntu 20.04. 2 LTS
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
argparse 1.4.0 pypi_0 pypi
batchgenerators 0.21 pypi_0 pypi
blas 1.0 mkl
bzip2 1.0.8 h7b6447c_0
ca-certificates 2021.5.25 h06a4308_1
certifi 2021.5.30 py38h06a4308_0
chardet 4.0.0 pypi_0 pypi
cudatoolkit 11.1.74 h6bb024c_0 nvidia
cycler 0.10.0 pypi_0 pypi
decorator 4.4.2 pypi_0 pypi
dicom2nifti 2.3.0 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
freetype 2.10.4 h5ab3b9f_0
future 0.18.2 pypi_0 pypi
gmp 6.2.1 h2531618_2
gnutls 3.6.15 he1e5248_0
hiddenlayer 0.2 pypi_0 pypi
idna 2.10 pypi_0 pypi
imageio 2.9.0 pypi_0 pypi
intel-openmp 2021.2.0 h06a4308_610
joblib 1.0.1 pypi_0 pypi
jpeg 9b h024ee3a_2
kiwisolver 1.3.1 pypi_0 pypi
lame 3.100 h7b6447c_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.35.1 h7274673_9
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libiconv 1.15 h63c8f33_5
libidn2 2.3.1 h27cfd23_0
libpng 1.6.37 hbc83047_0
libstdcxx-ng 9.3.0 hd4cf53a_17
libtasn1 4.16.0 h27cfd23_0
libtiff 4.2.0 h85742a9_0
libunistring 0.9.10 h27cfd23_0
libuv 1.40.0 h7b6447c_0
libwebp-base 1.2.0 h27cfd23_0
linecache2 1.0.0 pypi_0 pypi
lz4-c 1.9.3 h2531618_0
matplotlib 3.4.2 pypi_0 pypi
medpy 0.4.0 pypi_0 pypi
mkl 2021.2.0 h06a4308_296
mkl-service 2.3.0 py38h27cfd23_1
mkl_fft 1.3.0 py38h42c9631_2
mkl_random 1.2.1 py38ha9443f7_2
ncurses 6.2 he6710b0_1
nettle 3.7.3 hbbd107a_1
networkx 2.5.1 pypi_0 pypi
nibabel 3.2.1 pypi_0 pypi
ninja 1.10.2 hff7bd54_1
nnunet 1.6.6 dev_0 <develop>
numpy 1.20.2 py38h2d18471_0
numpy-base 1.20.2 py38hfae3a4d_0
olefile 0.46 py_0
openh264 2.1.0 hd408876_0
openssl 1.1.1k h27cfd23_0
packaging 20.9 pypi_0 pypi
pandas 1.2.5 pypi_0 pypi
pillow 8.2.0 py38he98fc37_0
pip 21.1.2 py38h06a4308_0
pydicom 2.1.2 pypi_0 pypi
pyparsing 2.4.7 pypi_0 pypi
python 3.8.10 h12debd9_8
python-dateutil 2.8.1 pypi_0 pypi
pytorch 1.9.0 py3.8_cuda11.1_cudnn8.0.5_0 pytorch
pytz 2021.1 pypi_0 pypi
pywavelets 1.1.1 pypi_0 pypi
readline 8.1 h27cfd23_0
requests 2.25.1 pypi_0 pypi
scikit-image 0.18.1 pypi_0 pypi
scikit-learn 0.24.2 pypi_0 pypi
scipy 1.7.0 pypi_0 pypi
setuptools 52.0.0 py38h06a4308_0
simpleitk 2.0.2 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_0
sklearn 0.0 pypi_0 pypi
sqlite 3.35.4 hdfb4753_0
threadpoolctl 2.1.0 pypi_0 pypi
tifffile 2021.6.14 pypi_0 pypi
tk 8.6.10 hbc83047_0
torchaudio 0.9.0 py38 pytorch
torchvision 0.10.0 py38_cu111 pytorch
tqdm 4.61.1 pypi_0 pypi
traceback2 1.4.0 pypi_0 pypi
typing_extensions 3.7.4.3 pyha847dfd_0
unittest2 1.1.0 pypi_0 pypi
urllib3 1.26.5 pypi_0 pypi
wheel 0.36.2 pyhd3eb1b0_0
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.9 haebb681_0 These are the packages I installed for nnUnet environment. I am suspecting medpy version is not in sync. May I get some suggestions from you? Thank you for your effort.
the medpy version is the same on my working env. simple itk as well. so what i would do is to double check couple of things. Can you open case_00038.nii.gz file separately from kits21 repo? Can you open it when its in nnUNet_raw folder? do those file with errors end up in preprocessing folder?
@trofimova No I can't open the 00038 file separately and also in raw folder. I am using ITK-snap software to open it, but it says format NiFTi can not read the image. My preprocessing folder is now empty since I meet this problem during preprocessing. Should I download the images again?
i would say yes. if you still have the problem with opening the error files separately, please let us and @neheller know.
If the problem is just with one case, you can delete that image file and then re-run get_imaging.py. it should re-download only the image that is missing.
@trofimova @neheller Oh yes, I have tried deleted case 38th image and then re download it, but the same problem of beubg unable to read the image occurs.
I found out that from number 6th image, the file size is extremely small. I am not sure why this happen though, it just didn't download properly.
That's odd, I just ran it and the files came down normally for me. Did the case you re-downloaded come down properly?
Yeah it looks like your download failed. Those images are supposed to be way larger. Can you delete all the small images and try redownloading? Do you have enough disk space available?
Yes, I have done it, I think it was due to download fail, since the preprocessing takes long time, should I just focus on some of cases and train?
No it only makes sense to use all images. Give it time :-)
@FabianIsensee When preprocessing stops because of connection close, when I restart the preprocessing, will the machine save the previous preprocessed images?
Preprocessing is currently always done from scratch. We could change that in the future but unfortunately that's how it is now. Please use -tl 2 and -tf 1 in an attempt to get it working :-) This needs a lot of RAM, unfortunately
Hey @mincheoree, were you able to retrieve the full dataset?
@neheller Oh yes I am able to solve the download problem after I re-downloaded the dataset. Thanks for your feedbacks!
Great!