nbclient.exceptions.DeadKernelError: Kernel died in ./modules/3d_image_transforms.ipynb
KumoLiu opened this issue · 7 comments
[2024-04-24T07:17:21.826Z] + ./runner.sh -t modules/3d_image_transforms.ipynb
[2024-04-24T07:17:21.826Z] -and -wholename ./modules/3d_image_transforms.ipynb
[2024-04-24T07:17:22.082Z] Notebook files to be tested:
[2024-04-24T07:17:22.082Z] ./modules/3d_image_transforms.ipynb
[2024-04-24T07:17:22.082Z] Running ./modules/3d_image_transforms.ipynb
[2024-04-24T07:17:22.082Z] Checking PEP8 compliance...
[2024-04-24T07:17:23.011Z] Running notebook...
[2024-04-24T07:17:31.108Z] MONAI version: 1.3.1rc5+10.g07a78d20
[2024-04-24T07:17:31.108Z] Numpy version: 1.24.4
[2024-04-24T07:17:31.108Z] Pytorch version: 2.3.0a0+40ec155e58.nv24.03
[2024-04-24T07:17:31.108Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2024-04-24T07:17:31.108Z] MONAI rev id: 07a78d20b30ab4abe8fd776063ab6903439df264
[2024-04-24T07:17:31.108Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2024-04-24T07:17:31.108Z]
[2024-04-24T07:17:31.108Z] Optional dependencies:
[2024-04-24T07:17:31.108Z] Pytorch Ignite version: 0.4.11
[2024-04-24T07:17:31.108Z] ITK version: 5.3.0
[2024-04-24T07:17:31.108Z] Nibabel version: 5.2.1
[2024-04-24T07:17:31.108Z] scikit-image version: 0.22.0
[2024-04-24T07:17:31.108Z] scipy version: 1.12.0
[2024-04-24T07:17:31.108Z] Pillow version: 10.2.0
[2024-04-24T07:17:31.108Z] Tensorboard version: 2.16.2
[2024-04-24T07:17:31.108Z] gdown version: 4.7.3
[2024-04-24T07:17:31.108Z] TorchVision version: 0.18.0a0
[2024-04-24T07:17:31.108Z] tqdm version: 4.66.2
[2024-04-24T07:17:31.108Z] lmdb version: 1.4.1
[2024-04-24T07:17:31.108Z] psutil version: 5.9.4
[2024-04-24T07:17:31.108Z] pandas version: 1.5.3
[2024-04-24T07:17:31.108Z] einops version: 0.7.0
[2024-04-24T07:17:31.108Z] transformers version: 4.40.1
[2024-04-24T07:17:31.108Z] mlflow version: 2.11.3
[2024-04-24T07:17:31.108Z] pynrrd version: 1.0.0
[2024-04-24T07:17:31.108Z] clearml version: 1.15.1
[2024-04-24T07:17:31.108Z]
[2024-04-24T07:17:31.108Z] For details about installing the optional dependencies, please visit:
[2024-04-24T07:17:31.108Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2024-04-24T07:17:31.109Z]
[2024-04-24T07:17:32.477Z] papermill --progress-bar --log-output -k python3
[2024-04-24T07:17:32.732Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2024-04-24T07:17:32.732Z] warnings.warn(
[2024-04-24T07:18:27.240Z]
Executing: 0%| | 0/44 [00:00<?, ?cell/s]
Executing: 2%|▏ | 1/44 [00:00<00:37, 1.14cell/s]
Executing: 9%|▉ | 4/44 [00:09<01:35, 2.39s/cell]
Executing: 14%|█▎ | 6/44 [00:15<01:42, 2.70s/cell]
Executing: 23%|██▎ | 10/44 [00:49<03:24, 6.01s/cell]
Executing: 41%|████ | 18/44 [00:50<01:01, 2.37s/cell]
Executing: 48%|████▊ | 21/44 [00:51<00:42, 1.83s/cell]
Executing: 50%|█████ | 22/44 [00:51<00:36, 1.68s/cell]
Executing: 66%|██████▌ | 29/44 [00:51<00:12, 1.18cell/s]
Executing: 73%|███████▎ | 32/44 [00:52<00:08, 1.43cell/s]
Executing: 77%|███████▋ | 34/44 [00:52<00:05, 1.67cell/s]
Executing: 86%|████████▋ | 38/44 [00:53<00:02, 2.19cell/s][monai-notebooks-v100-jenkins-monai-notebooks-1191-2m893-mzqr3:2014 :0:2014] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7f0859c97f68)
[2024-04-24T07:18:27.497Z] Kernel died while waiting for execute reply.
[2024-04-24T07:18:27.497Z]
Executing: 95%|█████████▌| 42/44 [00:54<00:00, 2.56cell/s]
Executing: 95%|█████████▌| 42/44 [00:54<00:02, 1.30s/cell]
[2024-04-24T07:18:27.497Z] Traceback (most recent call last):
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 254, in papermill
[2024-04-24T07:18:27.497Z] execute_notebook(
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 119, in execute_notebook
[2024-04-24T07:18:27.497Z] nb = papermill_engines.execute_notebook_with_engine(
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/papermill/engines.py", line 48, in execute_notebook_with_engine
[2024-04-24T07:18:27.497Z] return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/papermill/engines.py", line 365, in execute_notebook
[2024-04-24T07:18:27.497Z] cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/papermill/engines.py", line 434, in execute_managed_notebook
[2024-04-24T07:18:27.497Z] return PapermillNotebookClient(nb_man, **final_kwargs).execute()
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/papermill/clientwrap.py", line 45, in execute
[2024-04-24T07:18:27.497Z] self.papermill_execute_cells()
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/papermill/clientwrap.py", line 72, in papermill_execute_cells
[2024-04-24T07:18:27.497Z] self.execute_cell(cell, index)
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/jupyter_core/utils/__init__.py", line 165, in wrapped
[2024-04-24T07:18:27.497Z] return loop.run_until_complete(inner)
[2024-04-24T07:18:27.497Z] File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
[2024-04-24T07:18:27.497Z] return future.result()
[2024-04-24T07:18:27.497Z] File "/usr/local/lib/python3.10/dist-packages/nbclient/client.py", line 1009, in async_execute_cell
[2024-04-24T07:18:27.497Z] raise DeadKernelError("Kernel died") from None
[2024-04-24T07:18:27.497Z] nbclient.exceptions.DeadKernelError: Kernel died
It looks to have died on the second last cell:
deformed_data_dict = rand_elastic(data_dict)
print(f"image shape: {deformed_data_dict['image'].shape}")
image, label = deformed_data_dict["image"][0], deformed_data_dict["label"][0]
plt.figure("visualise", (8, 4))
plt.subplot(1, 2, 1)
plt.title("image")
plt.imshow(image[:, :, 5], cmap="gray")
plt.subplot(1, 2, 2)
plt.title("label")
plt.imshow(label[:, :, 5])
plt.show()I can't tell from that if it's the elastic deformation or the matplotlib calls that cause the crash, it could be either if the environment isn't right and some compiled code caused a crash.
Would be worth running with CUDA_LAUNCH_BLOCKING="1" if the test is running pre-processing on the GPU
Would be worth running with CUDA_LAUNCH_BLOCKING="1" if the test is running pre-processing on the GPU
I don't think the transforms are run on the GPU. You can see in this tutorial.
https://github.com/Project-MONAI/tutorials/blob/main/modules/3d_image_transforms.ipynb
Would be worth running with CUDA_LAUNCH_BLOCKING="1" if the test is running pre-processing on the GPU
I don't think the transforms are run on the GPU. You can see in this tutorial. https://github.com/Project-MONAI/tutorials/blob/main/modules/3d_image_transforms.ipynb
There may be some interaction anyway with the CUDA components in Pytorch so it's worth trying. Are there any other environment variables to set to enhance debug output? We don't have much else to go on since we can't replicate the issue locally.
tried with CUDA_LAUNCH_BLOCKING=1, the same error occurred.
Step to reproduce:
docker pull nvcr.io/nvidia/pytorch:24.03-py3
docker run ...
# install monai
git clone https://github.com/Project-MONAI/MONAI.git
python -m pip install --upgrade pip wheel
python -m pip install -r requirements-dev.txt
BUILD_MONAI=0 python setup.py develop
# install tutorial
git clone https://github.com/Project-MONAI/tutorials.git
python -m pip install -r requirements.txt; python -m pip list
# run notebook
CUDA_VISIBLE_DEVICES=0 CUDA_LAUNCH_BLOCKING=1 ./runner.sh -t modules/3d_image_transforms.ipynb
Next thing to try is enabling fault handling: https://docs.python.org/3/library/faulthandler.html Add PYTHONFAULTHANDLER=1 to the command line and see if we get a stack trace when the fault happens. If this doesn't work it's a matter of tracing the code somehow to see which line is the last to execute before a segfault happens. That's painful but we can possibly find a tool to do that cleanly.
I traced the error back to the Convolution in the Gaussian filter, which is used by the Rand3DElasticd transform. I suspect this issue is related to a previous bug I encountered. I guess PyTorch 24.03 container may not include the commit mentioned in the ticket, since the code runs successfully under PyTorch's nightly build. I will confirm it today.