Failed to run inference in MONAI Label Server - Slicer
annkapopp opened this issue · 9 comments
Hi,
I trained DeepEdit on a custom dataset and run the inference on the MONAI server. That has worked so far. When I load the model into Slicer, the segmentation is automatically loaded from the labels folder creating during the inference. When I click on Auto Segmentation -> run, there is the following error:
Failed to run inference in MONAI Label Server.
Traceback (most recent call last):
File "/home/ai/Slicer-5.4.0-linux-amd64/32291-linux-amd64-MONAILabel-git5ef153f-2023-11-03/lib/Slicer-5.5/qt-scripted-modules/MONAILabel.py", line 1532, in onClickSegmentation
self.updateSegmentationMask(result_file, labels)
File "/home/ai/Slicer-5.4.0-linux-amd64/32291-linux-amd64-MONAILabel-git5ef153f-2023-11-03/lib/Slicer-5.5/qt-scripted-modules/MONAILabel.py", line 1731, in updateSegmentationMask
labelImage = sitk.ReadImage(in_file)
File "/home/ai/Slicer-5.4.0-linux-amd64/lib/Python/lib/python3.9/site-packages/SimpleITK/extra.py", line 357, in ReadImage
return reader.Execute()
File "/home/ai/Slicer-5.4.0-linux-amd64/lib/Python/lib/python3.9/site-packages/SimpleITK/SimpleITK.py", line 6249, in Execute
return _SimpleITK.ImageFileReader_Execute(self)
RuntimeError: Exception thrown in SimpleITK ImageFileReader_Execute: /work/Stable/Slicer-0-build/ITK/Modules/IO/NRRD/src/itkNrrdImageIO.cxx:292:
itk::ERROR: NrrdImageIO(0x10345d70): ReadImageInformation: Error reading /tmp/Slicer-ai/slicer-monai-label2023-11-08_09+00+18.204/tmpkjmn3rsm.nrrd:
[nrrd] nrrdLoad: trouble reading "/tmp/Slicer-ai/slicer-monai-label2023-11-08_09+00+18.204/tmpkjmn3rsm.nrrd"
[nrrd] nrrdRead: trouble
[nrrd] _nrrdRead: trouble reading NRRD file
[nrrd] _nrrdFormatNRRD_read: trouble parsing space directions info |(0,703125,0,0) (0,0,703125,0) (0,0,4)|
[nrrd] _nrrdReadNrrdParse_space_directions: trouble getting space vector 1 of 3
[nrrd] _nrrdSpaceVectorParse: space dimension is 3, but seem to have 4 coefficients
The same error details are displayed when placing SmartEdit / Deepgrow landmarks.
Expected behavior
Auto segmentation is executed and it can be adjusted by placing landmarks without any error occuring.
Environment
================================
Printing MONAI config...
================================
MONAI version: 1.3.0
Numpy version: 1.24.4
Pytorch version: 2.1.0+cu121
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 865972f7a791bf7b42efbcd87c8402bd865b329e
MONAI __file__: /home/<username>/.local/lib/python3.8/site-packages/mon<username>/__init__.py
Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.3.0
Nibabel version: 3.2.1
scikit-image version: 0.21.0
scipy version: 1.10.1
Pillow version: 10.0.1
Tensorboard version: 2.14.0
gdown version: 4.7.1
TorchVision version: 0.16.0+cu121
tqdm version: 4.58.0
lmdb version: 1.4.1
psutil version: 5.9.6
pandas version: 1.2.2
einops version: 0.7.0
transformers version: 4.21.3
mlflow version: 2.8.0
pynrrd version: 1.0.0
clearml version: 1.13.2rc2
For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 20.04.6 LTS
Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.29
Processor: x86_64
Machine: x86_64
Python version: 3.8.10
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 4
Num logical CPUs: 8
Num usable CPUs: 8
CPU usage (%): [25.3, 43.2, 55.1, 37.4, 21.6, 49.9, 35.7, 25.2]
CPU freq. (MHz): 1754
Load avg. in last 1, 5, 15 mins (%): [27.8, 26.4, 25.9]
Disk usage (%): 86.6
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 62.7
Available memory (GB): 50.7
Used memory (GB): 10.2
================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 12.1
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 8902
Current device: 0
Library compiled for CUDA architectures: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
GPU 0 Name: NVIDIA TITAN X (Pascal)
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 28
GPU 0 Total memory (GB): 11.9
GPU 0 CUDA capability (maj.min): 6.1
I really hope that somebody can help me understand the error details and has an idea how to solve it. Thank you!
Hi @annkapopp,
It seems the file is corrupted. Can you please share the MONAI Label logs? Also, did you try loading the volume directly to 3DSlicer? Did it work?
Let us know,
Hi @diazandr3s,
here are the logs:
[2023-11-09 09:15:49,062] [209977] [MainThread] [INFO] (monailabel.endpoints.infer:171) - Infer Request: {'model': 'deepedit_seg', 'image': '3Dircadb1.19_0000', 'device': 'NVIDIA TITAN X (Pascal)', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'}
[2023-11-09 09:15:49,062] [209977] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:280) - Infer Request (final): {'device': 'cuda', 'model': 'deepedit_seg', 'image': '/mnt/DATA1/nnUNetv2/nnUNet_raw/Dataset001_3D-IRCADb-01/imagesTs/3Dircadb1.19_0000.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'A DeepEdit model for volumetric (3D) segmentation over 3D Images'}
[2023-11-09 09:15:49,063] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s)
[2023-11-09 09:15:49,063] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path']
[2023-11-09 09:15:49,944] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 0.8799; image: torch.Size([512, 512, 124])(torch.float32)
[2023-11-09 09:15:49,944] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0002; image: torch.Size([1, 512, 512, 124])(torch.float32)
[2023-11-09 09:15:49,972] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Orientationd): Time: 0.0274; image: torch.Size([1, 512, 512, 124])(torch.float32)
[2023-11-09 09:15:50,144] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.1714; image: torch.Size([1, 512, 512, 124])(torch.float32)
[2023-11-09 09:15:50,339] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Resized): Time: 0.1947; image: torch.Size([1, 128, 128, 128])(torch.float32)
[2023-11-09 09:15:50,355] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (DiscardAddGuidanced): Time: 0.0163; image: torch.Size([3, 128, 128, 128])(torch.float32)
[2023-11-09 09:15:50,360] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.0039; image: torch.Size([3, 128, 128, 128])(torch.float32)
[2023-11-09 09:15:50,360] [209977] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:504) - Inferer:: cuda => SimpleInferer => {}
[2023-11-09 09:15:50,360] [209977] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: /mnt/DATA1/DeepEdit/Dataset001_3D-IRCADb-01/fold0/pretrained_deepedit_dynunet.pt
[2023-11-09 09:15:50,819] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - INV - Run Transform(s)
[2023-11-09 09:15:50,819] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - INV - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2023-11-09 09:15:50,824] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (Resized): Time: 0.0035; image: torch.Size([2, 512, 512, 124])(torch.float32); pred: torch.Size([2, 128, 128, 128])(torch.float32)
[2023-11-09 09:15:50,829] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (Orientationd): Time: 0.0049; image: torch.Size([2, 512, 512, 124])(torch.float32); pred: torch.Size([2, 128, 128, 128])(torch.float32)
[2023-11-09 09:15:50,830] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s)
[2023-11-09 09:15:50,830] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2023-11-09 09:15:50,831] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0001; image: torch.Size([3, 128, 128, 128])(torch.float32); pred: torch.Size([2, 512, 512, 124])(torch.float32)
[2023-11-09 09:15:50,831] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.0003; image: torch.Size([3, 128, 128, 128])(torch.float32); pred: torch.Size([2, 512, 512, 124])(torch.float32)
[2023-11-09 09:15:50,837] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.006; image: torch.Size([3, 128, 128, 128])(torch.float32); pred: torch.Size([1, 512, 512, 124])(torch.float32)
[2023-11-09 09:15:50,838] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (SqueezeDimd): Time: 0.0002; image: torch.Size([3, 128, 128, 128])(torch.float32); pred: torch.Size([512, 512, 124])(torch.float32)
[2023-11-09 09:15:50,939] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (ToNumpyd): Time: 0.1004; image: torch.Size([3, 128, 128, 128])(torch.float32); pred: (512, 512, 124)(float32)
[2023-11-09 09:15:50,939] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0001; image: torch.Size([3, 128, 128, 128])(torch.float32); pred: (512, 512, 124)(float32)
[2023-11-09 09:15:51,872] [209977] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (GetCentroidsd): Time: 0.9331; image: torch.Size([3, 128, 128, 128])(torch.float32); pred: (512, 512, 124)(float32)
[2023-11-09 09:15:51,873] [209977] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:604) - Writing Result...
[2023-11-09 09:15:51,873] [209977] [MainThread] [INFO] (monailabel.transform.writer:196) - Result ext: .nrrd; write_to_file: True; dtype: uint8
[2023-11-09 09:15:52,325] [209977] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 3.2627; Pre: 1.2971; Inferer: 0.4535; Invert: 0.0160; Post: 1.0434; Write: 0.4521
[2023-11-09 09:15:52,325] [209977] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:370) - Result File: /tmp/tmph3c2avnt.nrrd
[2023-11-09 09:15:52,325] [209977] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['label_names', 'latencies', 'centroids']
I can load the image volume and the segmentation from the inference on the label server into Slicer and they are both displayed correctly. When I only load the image volume, the auto segmentation does not do anything and the landmark button is greyed out.
Hi @annkapopp,
Thanks for sharing the logs.
I don't see anything strange from the logs. Inference ran smoothly and the mask was generated. Do you know if this is happening consistently? Can you please try with another volume?
Hey @diazandr3s ,
yes, this is happening consistently with every volume in the dataset.
Hi @annkapopp,
This is strange - I don't see any issue from the MONAI Label side
MONAI Label should work with any Slicer version. But to discard issues, can you please try with the Preview Release of 3DSlicer (version 5.5.0)?
Hi, @diazandr3s,
well, that is strange.. It shows the same errors with Slicer 5.5.0...
Hi @annkapopp,
I'm trying to understand what's the issue here as I don't see anything wrong with the MONAI Label logs.
Can you please try with a publicly available dataset (i.e. Medical Segmentation Decathlon)? I suspect this issue may come from the dataset.
Otherwise, if you still see the grey button in Slicer, you may find this thread useful: #852 (comment)
Please let us know
Hi @diazandr3s,
I moved everything to a different computer and somehow, it does work now! But I do not really know why.
But thank you very much for your time and for trying to help me!
Hi @annkapopp,
Thanks for letting us know.