New directory is not created while using flag --no-train

Question

New directory is not created while using flag --no-train

Adapter668 opened this issue 2 years ago · 4 comments

Is there an existing issue for this?

I have searched the existing issues

Bug summary

While using flag --no-train there is an error that path does not exist. I tried also using local_weights_path, but still the same error.

Code for reproduction

CUDA_VISIBLE_DEVICES=5 python InnerEye/ML/runner.py --model=DynamicUnetBones --no-train --local_weights_path=outputs/2023-05-19T071816Z_DynamicUnetBones/checkpoints/last.ckpt

Actual outcome

Traceback (most recent call last):
File "InnerEye/ML/runner.py", line 467, in
main()
File "InnerEye/ML/runner.py", line 461, in main
run(project_root=fixed_paths.repository_root_directory(),
File "InnerEye/ML/runner.py", line 457, in run
return runner.run()
File "InnerEye/ML/runner.py", line 220, in run
self.run_in_situ(azure_run_info)
File "InnerEye/ML/runner.py", line 412, in run_in_situ
self.ml_runner.run()
File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/ML/run_ml.py", line 378, in run
self.run_inference_for_lightning_models(checkpoint_paths_for_testing)
File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/ML/run_ml.py", line 469, in run_inference_for_lightning_models
with change_working_directory(self.container.outputs_folder):
File "/home/.conda/envs/InnerEye3/lib/python3.8/contextlib.py", line 113, in enter
return next(self.gen)
File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/Common/common_util.py", line 436, in change_working_directory
os.chdir(new_path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/outputs/2023-05-22T073814Z_DynamicUnetBones'

Error messages

FileNotFoundError: [Errno 2] No such file or directory: '/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/outputs/2023-05-22T073814Z_DynamicUnetBones'

Expected outcome

It should run predictions on test data and save the result either on the local_weights_path directory or the new path (doesn't matter for me)

System info

OS:
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

INNEREYE COMMIT HASH:
commit 2877002 (HEAD -> main, origin/main, origin/HEAD)

CONDA ENV:
name: InnerEye3
channels:

pytorch
defaults
dependencies:
_libgcc_mutex=0.1=main
_openmp_mutex=5.1=1_gnu
blas=1.0=mkl
blosc=1.21.0=h4ff587b_1
bzip2=1.0.8=h7b6447c_0
ca-certificates=2022.07.19=h06a4308_0
certifi=2022.9.24=py38h06a4308_0
cudatoolkit=11.3.1=h2bc3f7f_2
ffmpeg=4.2.2=h20bf706_0
freetype=2.11.0=h70c0345_0
giflib=5.2.1=h7b6447c_0
gmp=6.2.1=h295c915_3
gnutls=3.6.15=he1e5248_0
intel-openmp=2021.4.0=h06a4308_3561
jpeg=9e=h7f8727e_0
lame=3.100=h7b6447c_0
lcms2=2.12=h3be6417_0
ld_impl_linux-64=2.38=h1181459_1
lerc=3.0=h295c915_0
libdeflate=1.8=h7f8727e_5
libffi=3.3=he6710b0_2
libgcc-ng=11.2.0=h1234567_1
libgomp=11.2.0=h1234567_1
libidn2=2.3.2=h7f8727e_0
libopus=1.3.1=h7b6447c_0
libpng=1.6.37=hbc83047_0
libstdcxx-ng=11.2.0=h1234567_1
libtasn1=4.16.0=h27cfd23_0
libtiff=4.4.0=hecacb30_0
libunistring=0.9.10=h27cfd23_0
libuv=1.40.0=h7b6447c_0
libvpx=1.7.0=h439df22_0
libwebp=1.2.4=h11a3e52_0
libwebp-base=1.2.4=h5eee18b_0
lz4-c=1.9.3=h295c915_1
mkl=2021.4.0=h06a4308_640
mkl-service=2.4.0=py38h7f8727e_0
mkl_fft=1.3.1=py38hd3c417c_0
mkl_random=1.2.2=py38h51133e4_0
ncurses=6.3=h5eee18b_3
nettle=3.7.3=hbbd107a_1
openh264=2.1.1=h4ff587b_0
openssl=1.1.1q=h7f8727e_0
pip=20.1.1=py38_1
python=3.8.3=hcff3b4d_2
python-blosc=1.7.0=py38h7b6447c_0
pytorch=1.10.0=py3.8_cuda11.3_cudnn8.2.0_0
pytorch-mutex=1.0=cuda
readline=8.1.2=h7f8727e_1
setuptools=63.4.1=py38h06a4308_0
sqlite=3.39.3=h5082296_0
tk=8.6.12=h1ccaba5_0
torchvision=0.11.1=py38_cu113
typing_extensions=4.3.0=py38h06a4308_0
wheel=0.37.1=pyhd3eb1b0_0
x264=1!157.20191217=h7b6447c_0
xz=5.2.6=h5eee18b_0
zlib=1.2.12=h5eee18b_3
zstd=1.5.2=ha4553b6_0
pip:
- absl-py==1.2.0
- adal==1.2.7
- aiofiles==23.1.0
- aiohttp==3.8.3
- aiosignal==1.2.0
- alabaster==0.7.12
- alembic==1.8.1
- ansiwrap==0.8.4
- applicationinsights==0.11.10
- argon2-cffi==21.3.0
- argon2-cffi-bindings==21.2.0
- argparse==1.4.0
- async-timeout==4.0.2
- attrs==22.1.0
- azure-common==1.1.28
- azure-core==1.26.0
- azure-graphrbac==0.61.1
- azure-identity==1.7.0
- azure-mgmt-authorization==0.61.0
- azure-mgmt-containerregistry==10.0.0
- azure-mgmt-core==1.3.2
- azure-mgmt-datafactory==1.1.0
- azure-mgmt-keyvault==9.3.0
- azure-mgmt-resource==12.1.0
- azure-mgmt-storage==11.2.0
- azure-storage-blob==12.6.0
- azureml-automl-core==1.36.1
- azureml-core==1.36.0.post2
- azureml-dataprep==2.24.4
- azureml-dataprep-native==38.0.0
- azureml-dataprep-rslex==2.0.3
- azureml-dataset-runtime==1.36.0
- azureml-mlflow==1.36.0
- azureml-pipeline==1.36.0
- azureml-pipeline-core==1.36.0
- azureml-pipeline-steps==1.36.0
- azureml-sdk==1.36.0
- azureml-telemetry==1.36.0
- azureml-tensorboard==1.36.0
- azureml-train-automl-client==1.36.0
- azureml-train-core==1.36.0
- azureml-train-restclients-hyperdrive==1.36.0
- babel==2.10.3
- backcall==0.2.0
- backports-tempfile==1.0
- backports-weakref==1.0.post1
- batchgenerators==0.25
- beautifulsoup4==4.11.1
- black==22.10.0
- bleach==5.0.1
- cachetools==4.2.4
- cffi==1.15.1
- charset-normalizer==2.1.1
- click==8.1.3
- cloudpickle==1.6.0
- colorama==0.4.5
- commonmark==0.9.1
- conda-merge==0.1.5
- contextlib2==21.6.0
- coverage==6.5.0
- cryptography==3.3.2
- cycler==0.11.0
- databricks-cli==0.17.3
- dataclasses-json==0.5.2
- debugpy==1.6.3
- decorator==5.1.1
- defusedxml==0.7.1
- deprecated==1.2.13
- dicom2nifti==2.4.8
- dill==0.3.6
- distro==1.8.0
- docker==4.3.1
- docutils==0.18.1
- dotnetcore2==2.1.23
- entrypoints==0.4
- execnet==1.9.0
- fastjsonschema==2.16.2
- fastmri==0.2.0
- flake8==3.8.3
- flask==2.2.2
- frozenlist==1.3.1
- fsspec==2022.8.2
- furo==2022.6.21
- fusepy==3.0.1
- future==0.18.3
- gitdb==4.0.9
- gitpython==3.1.7
- google-auth==1.35.0
- google-auth-oauthlib==0.4.6
- gputil==1.4.0
- greenlet==1.1.3.post0
- grpcio==1.49.1
- gunicorn==20.1.0
- h5py==2.10.0
- hi-ml==0.2.5
- hi-ml-azure==0.2.5
- humanize==4.4.0
- idna==3.4
- imageio==2.15.0
- imagesize==1.4.1
- importlib-metadata==5.0.0
- importlib-resources==5.10.0
- iniconfig==1.1.1
- innereye-dicom-rt==1.1.0
- ipykernel==6.16.0
- ipython==7.31.1
- ipython-genutils==0.2.0
- ipywidgets==8.0.2
- isodate==0.6.1
- itsdangerous==2.1.2
- jedi==0.18.2
- jeepney==0.8.0
- jinja2==3.0.2
- jmespath==0.10.0
- joblib==0.16.0
- jsonpickle==2.2.0
- jsonschema==4.16.0
- jupyter==1.0.0
- jupyter-client==6.1.5
- jupyter-console==6.4.4
- jupyter-core==4.11.1
- jupyterlab-pygments==0.2.2
- jupyterlab-widgets==3.0.3
- kiwisolver==1.4.4
- lightning-bolts==0.4.0
- linecache2==1.0.0
- llvmlite==0.34.0
- mako==1.2.3
- markdown==3.4.1
- markdown-it-py==2.1.0
- markupsafe==2.1.1
- marshmallow==3.18.0
- marshmallow-enum==1.5.1
- matplotlib==3.3.0
- matplotlib-inline==0.1.6
- mccabe==0.6.1
- mdit-py-plugins==0.3.1
- mdurl==0.1.2
- medpy==0.4.0
- mistune==2.0.4
- mlflow==1.23.1
- mlflow-skinny==1.29.0
- monai==0.6.0
- more-itertools==8.14.0
- msal==1.20.0
- msal-extensions==0.3.1
- msrest==0.7.1
- msrestazure==0.6.4
- multidict==6.0.2
- mypy==0.910
- mypy-extensions==0.4.3
- myst-parser==0.18.0
- nbclient==0.7.0
- nbconvert==7.2.1
- nbformat==5.7.0
- ndg-httpsclient==0.5.1
- nest-asyncio==1.5.6
- networkx==2.8.7
- nibabel==4.0.2
- nnunet==1.7.0
- notebook==6.4.12
- numba==0.51.2
- numpy==1.19.1
- oauthlib==3.2.1
- opencv-python-headless==4.5.1.48
- packaging==21.3
- pandas==1.1.0
- pandocfilters==1.5.0
- papermill==2.2.2
- param==1.9.3
- parso==0.8.3
- pathspec==0.10.1
- pathtools==0.1.2
- pexpect==4.8.0
- pickleshare==0.7.5
- pillow==9.1.1
- pkgutil-resolve-name==1.3.10
- platformdirs==2.5.2
- pluggy==0.13.1
- portalocker==2.5.1
- prometheus-client==0.14.1
- prometheus-flask-exporter==0.20.3
- prompt-toolkit==3.0.38
- protobuf==3.20.1
- psutil==5.7.2
- ptyprocess==0.7.0
- py==1.11.0
- pyarrow==3.0.0
- pyasn1==0.4.8
- pyasn1-modules==0.2.8
- pycodestyle==2.6.0
- pycparser==2.21
- pydeprecate==0.3.2
- pydicom==2.0.0
- pyflakes==2.2.0
- pygments==2.15.1
- pyjwt==2.4.0
- pynndescent==0.5.7
- pyopenssl==20.0.1
- pyparsing==3.0.9
- pyrsistent==0.18.1
- pysocks==1.7.1
- pytest==6.0.1
- pytest-cov==2.10.1
- pytest-forked==1.3.0
- pytest-xdist==1.34.0
- python-dateutil==2.8.2
- python-gdcm==3.0.22
- pytorch-lightning==1.6.5
- pytz==2022.4
- pywavelets==1.4.1
- pyyaml==6.0
- pyzmq==24.0.1
- qtconsole==5.3.2
- qtpy==2.2.1
- querystring-parser==1.2.4
- radio==0.1.0
- requests==2.28.1
- requests-oauthlib==1.3.1
- rich==10.13.0
- rpdb==0.1.6
- rsa==4.9
- ruamel-yaml==0.16.12
- ruamel-yaml-clib==0.2.6
- runstats==1.8.0
- scikit-image==0.17.2
- scikit-learn==0.23.2
- scipy==1.5.2
- seaborn==0.10.1
- secretstorage==3.3.3
- send2trash==1.8.0
- simpleitk==1.2.4
- six==1.15.0
- sklearn==0.0.post5
- smmap==5.0.0
- snowballstemmer==2.2.0
- soupsieve==2.3.2.post1
- sphinx==5.0.2
- sphinx-argparse==0.3.1
- sphinx-basic-ng==1.0.0b1
- sphinx-rtd-theme==1.0.0
- sphinxcontrib-applehelp==1.0.2
- sphinxcontrib-devhelp==1.0.2
- sphinxcontrib-htmlhelp==2.0.0
- sphinxcontrib-jsmath==1.0.1
- sphinxcontrib-qthelp==1.0.3
- sphinxcontrib-serializinghtml==1.1.5
- sqlalchemy==1.4.41
- sqlparse==0.4.3
- stopit==1.1.2
- stringcase==1.2.0
- tabulate==0.8.7
- tenacity==8.1.0
- tensorboard==2.3.0
- tensorboard-plugin-wit==1.8.1
- tensorboardx==2.1
- terminado==0.16.0
- textwrap3==0.9.2
- threadpoolctl==3.1.0
- tifffile==2022.10.10
- tinycss2==1.1.1
- toml==0.10.2
- tomli==2.0.1
- torchio==0.18.74
- torchmetrics==0.6.0
- tornado==6.2
- tqdm==4.64.1
- traceback2==1.4.0
- traitlets==5.9.0
- typing-inspect==0.8.0
- umap-learn==0.5.2
- unittest2==1.1.0
- urllib3==1.26.7
- wcwidth==0.2.6
- webencodings==0.5.1
- websocket-client==1.4.1
- werkzeug==2.2.2
- widgetsnbextension==4.0.3
- wrapt==1.14.1
- yacs==0.1.8
- yarl==1.8.1
- zipp==3.9.0
  prefix: /home/.conda/envs/InnerEye3

Answer 1 · 2023-05-22T20:22:50.000Z

Thanks @Adapter668 - this should be easily fixed, would you like to give it a try?
All you'd need to do is add self.container.outputs_folder.mkdir(parents=True, exist_ok=True) in the run_inference_for_lightning_models, before it calls change_working_directory(self.container.outputs_folder)

Answer 2 · 2023-05-24T05:14:33.000Z

Thanks, I'll try your solution :) I used workaround and added os.makedirs in common_util.py for now, but I guess it'll be nice in the future if the --no-train flag works without changing code

Answer 3 · 2023-05-30T08:27:37.000Z

Hi @Adapter668 , I was more thinking that you could create a PR with that change to integrate it for later use

Answer 4 · 2023-05-30T08:29:44.000Z

Yes, I know, I just need to have some time to test if it works fine and I'll create PR