ys-zong/MEDFAIR

Issues Installing Packages from `environments.yml` on macOS ARM and Ubuntu

Closed this issue · 2 comments

MASTER-WM commented

Description

I am experiencing several issues while trying to set up my development environment using an environments.yml file on both macOS ARM and Ubuntu. Below are the detailed steps and errors I encountered:

Steps to Reproduce

  1. Attempt to create the environment on macOS ARM using the environments.yml file:

    conda env create -n fair_benchmark -f environment.yml

  2. Encounter the following error:

    PackagesNotFoundError: The following packages are not available from current channels:
  - libstdcxx-ng
  - libgcc-ng
  - _libgcc_mutex

  3. Attempt to create the environment on Ubuntu using the environments.yml file:

    conda env create -n fair_benchmark -f environment.yml

  4. Encounter the following error:

    ERROR: No matching distribution found for oapackage==2.4.4

  5. Remove oapackage from the environments.yml file and attempt to create the environment again. Encounter the following error:

    ERROR: No matching distribution found for torch==1.10.2+cu111

  6. Remove torch from the environments.yml file and attempt to create the environment again. Encounter the following error:

    ERROR: No matching distribution found for torchvision==0.11.3+cu111

  7. Remove torchvision from the environments.yml file and attempt to create the environment again. Encounter the following error:

    ERROR: Cannot install -r /home/amir/PycharmProjects/Medfair/MEDFAIR/condaenv.3zil8mmr.requirements.txt (line 132) and tornado==4.5.3 because these package versions have conflicting dependencies.

  8. Attempt to install packages using pip. Some packages install successfully, but I encounter errors stating that packages are not installed when I try to use them.

Expected Behavior

The environment should be created successfully with all specified packages installed and functional.

Actual Behavior

Encountering multiple errors related to missing packages and conflicting dependencies.

System Information

  • Operating System:
    • macOS ARM
    • Ubuntu
  • Python Version: 3.8.12
  • Conda Version: 24.5.0
  • Pip Version: 24.1.1

environments.yml (MEDFAIR)

name: torch11
channels:
  - anaconda
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - backcall=0.2.0=py_0
  - ca-certificates=2020.10.14=0
  - certifi=2020.6.20=py38_0
  - ipython_genutils=0.2.0=py38_0
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - ncurses=6.3=h7f8727e_2
  - openssl=1.1.1m=h7f8727e_0
  - pexpect=4.8.0=py38_0
  - pickleshare=0.7.5=py38_1000
  - pip=21.2.4=py38h06a4308_0
  - python=3.8.12=h12debd9_0
  - readline=8.1.2=h7f8727e_1
  - setuptools=58.0.4=py38h06a4308_0
  - sqlite=3.37.0=hc218d9a_0
  - tk=8.6.11=h1ccaba5_0
  - wcwidth=0.2.5=py_0
  - xz=5.2.5=h7b6447c_0
  - zeromq=4.3.3=he6710b0_3
  - zlib=1.2.11=h7f8727e_4
  - pip:
    - absl-py==1.0.0
    - addict==2.4.0
    - albumentations==0.5.2
    - anyio==3.6.1
    - anykeystore==0.2
    - anyqt==0.1.1
    - argon2-cffi==21.3.0
    - argon2-cffi-bindings==21.2.0
    - arrow==1.2.2
    - asttokens==2.0.5
    - astunparse==1.6.3
    - attrs==21.4.0
    - autopep8==1.6.0
    - av==9.2.0
    - awscli==1.22.77
    - baycomp==1.0.2
    - bayesian-torch==0.2.1
    - beautifulsoup4==4.11.1
    - black==22.1.0
    - bleach==4.1.0
    - botocore==1.24.22
    - bottleneck==1.3.5
    - cachecontrol==0.12.11
    - cachetools==5.0.0
    - cffi==1.15.0
    - chardet==4.0.0
    - charset-normalizer==2.0.12
    - click==8.0.3
    - colorama==0.4.3
    - colorcet==3.0.0
    - commonmark==0.9.1
    - configargparse==1.5.3
    - cryptacular==1.6.2
    - cryptography==37.0.4
    - cycler==0.11.0
    - debugpy==1.5.1
    - decorator==5.1.1
    - defusedxml==0.7.1
    - deprecated==1.2.13
    - dicom2nifti==2.3.3
    - dictdiffer==0.9.0
    - diffdist==0.1
    - dill==0.3.4
    - docker-pycreds==0.4.0
    - docopt==0.6.2
    - docutils==0.15.2
    - dominate==2.6.0
    - efficientnet-pytorch==0.6.3
    - entrypoints==0.4
    - et-xmlfile==1.1.0
    - executing==0.8.2
    - face-alignment==1.3.5
    - filelock==3.7.0
    - fire==0.4.0
    - flatbuffers==2.0
    - fonttools==4.29.1
    - future==0.18.2
    - fvcore==0.1.5.post20220512
    - gast==0.5.3
    - gitdb==4.0.9
    - gitpython==3.1.27
    - google-auth==2.6.2
    - google-auth-oauthlib==0.4.6
    - google-pasta==0.2.0
    - greenlet==1.1.2
    - grpcio==1.44.0
    - h11==0.12.0
    - h5py==3.6.0
    - httpcore==0.15.0
    - httpx==0.23.0
    - humanize==3.14.0
    - hupper==1.10.3
    - idna==2.10
    - idx2numpy==1.2.3
    - imageio==2.16.0
    - imageio-ffmpeg==0.4.5
    - imgaug==0.4.0
    - importlib-metadata==4.11.3
    - importlib-resources==5.4.0
    - imutils==0.5.4
    - iopath==0.1.9
    - ipydatawidgets==4.2.0
    - itk==5.2.1.post1
    - itk-core==5.2.1.post1
    - itk-filtering==5.2.1.post1
    - itk-io==5.2.1.post1
    - itk-meshtopolydata==0.7.1
    - itk-numerics==5.2.1.post1
    - itk-registration==5.2.1.post1
    - itk-segmentation==5.2.1.post1
    - itkwidgets==0.32.1
    - jedi==0.17.2
    - jeepney==0.8.0
    - jinja2==3.0.3
    - jmespath==1.0.0
    - joblib==1.1.0
    - jsonpatch==1.32
    - jsonpickle==1.5.2
    - jsonpointer==2.2
    - jsonschema==4.4.0
    - keyring==23.7.0
    - keyrings-alt==4.1.1
    - kiwisolver==1.3.2
    - lesscpy==0.15.0
    - libclang==13.0.0
    - llvmlite==0.38.0
    - lmdb==1.3.0
    - lockfile==0.12.2
    - lpips==0.1.4
    - markdown==3.3.6
    - markupsafe==2.0.1
    - matplotlib==3.5.1
    - matplotlib-inline==0.1.3
    - mdai==0.11.0
    - mistune==0.8.4
    - mmcls==0.23.0
    - mmcv-full==1.4.2
    - mpi4py==3.1.3
    - msgpack==1.0.4
    - multicoretsne==0.1
    - munch==2.5.0
    - mypy-extensions==0.4.3
    - natsort==8.1.0
    - nbclient==0.5.10
    - nbconvert==6.4.2
    - nbformat==5.1.3
    - nest-asyncio==1.5.4
    - netcal==1.1.2
    - networkx==2.6.3
    - nibabel==3.2.1
    - nilearn==0.9.0
    - notebook==6.4.12
    - numba==0.55.1
    - numpy==1.22.0
    - nvidia-ml-py3==7.352.0
    - oapackage==2.4.4
    - oauthlib==3.2.0
    - opencv-contrib-python==4.5.5.64
    - opencv-python==4.5.5.62
    - opencv-python-headless==4.5.5.64
    - openpyxl==3.0.9
    - openslide-python==1.1.2
    - opentsne==0.6.2
    - opt-einsum==3.3.0
    - orange-canvas-core==0.1.27
    - orange-widget-base==4.17.0
    - orange3==3.32.0
    - packaging==21.3
    - pandas==1.4.0
    - pandocfilters==1.5.0
    - param==1.12.0
    - parameterized==0.8.1
    - parso==0.7.1
    - pastedeploy==2.1.1
    - pathspec==0.9.0
    - pathtools==0.1.2
    - patsy==0.5.2
    - pbkdf2==1.3
    - pillow==9.0.1
    - plaster==1.0
    - plaster-pastedeploy==0.7
    - platformdirs==2.5.0
    - ply==3.11
    - portalocker==2.4.0
    - pretrainedmodels==0.7.4
    - progressbar2==4.0.0
    - prometheus-client==0.13.1
    - promise==2.3
    - prompt-toolkit==3.0.27
    - protobuf==3.19.4
    - psutil==5.9.0
    - ptyprocess==0.7.0
    - pure-eval==0.2.2
    - py-cpuinfo==8.0.0
    - pyasn1==0.4.8
    - pyasn1-modules==0.2.8
    - pycodestyle==2.8.0
    - pycparser==2.21
    - pyct==0.4.8
    - pydicom==2.3.0
    - pyglet==1.5.23
    - pygments==2.11.2
    - pylibjpeg==1.4.0
    - pylibjpeg-libjpeg==1.3.1
    - pynndescent==0.5.7
    - pynvml==11.4.1
    - pyparsing==3.0.7
    - pyqt5==5.15.7
    - pyqt5-qt5==5.15.2
    - pyqt5-sip==12.11.0
    - pyqtgraph==0.12.3
    - pyqtwebengine==5.15.6
    - pyqtwebengine-qt5==5.15.2
    - pyramid==2.0
    - pyramid-mailer==0.15.1
    - pyrsistent==0.18.1
    - python-dateutil==2.8.2
    - python-gdcm==3.0.12
    - python-utils==3.1.0
    - python3-openid==3.2.0
    - pytorchvideo==0.1.5
    - pytz==2021.3
    - pywavelets==1.2.0
    - pyyaml==5.4.1
    - pyzmq==22.3.0
    - qasync==0.23.0
    - qimage2ndarray==1.9.0
    - qtconsole==5.2.2
    - qtpy==2.0.1
    - qudida==0.0.4
    - remote-ikernel==0.4.6
    - repoze-sendmail==4.4.1
    - requests==2.25.1
    - requests-oauthlib==1.3.1
    - retrying==1.3.3
    - rfc3986==1.5.0
    - rsa==4.7.2
    - s3transfer==0.5.2
    - sacred==0.8.2
    - scikit-image==0.19.2
    - scikit-learn==1.0.2
    - scikit-posthocs==0.7.0
    - scipy==1.7.3
    - seaborn==0.11.2
    - secretstorage==3.3.2
    - send2trash==1.8.0
    - sentry-sdk==1.5.6
    - serverfiles==0.3.1
    - setproctitle==1.2.2
    - shortuuid==1.0.8
    - simpleitk==2.1.1
    - six==1.16.0
    - sklearn==0.0
    - smmap==5.0.0
    - sniffio==1.2.0
    - soupsieve==2.3.2.post1
    - sqlalchemy==1.4.37
    - stack-data==0.1.4
    - statannot==0.2.3
    - statsmodels==0.13.2
    - strconv==0.4.2
    - tabulate==0.8.9
    - tensorboard==2.8.0
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.1
    - termcolor==1.1.0
    - terminado==0.13.1
    - testpath==0.5.0
    - tf-estimator-nightly==2.8.0.dev2021122109
    - thop==0.0.31-2005241907
    - threadpoolctl==3.1.0
    - tifffile==2022.2.9
    - toml==0.10.2
    - tomli==2.0.1
    - torch==1.10.2+cu111
    - torchfile==0.1.0
    - torchsampler==0.1.2
    - torchvision==0.11.3+cu111
    - tornado==4.5.3
    - tqdm==4.62.3
    - traitlets==5.1.1
    - traittypes==0.2.1
    - transaction==3.0.1
    - translationstring==1.4
    - typing-extensions==4.0.1
    - umap-learn==0.5.3
    - urllib3==1.26.8
    - velruse==1.1.1
    - venusian==3.0.0
    - visdom==0.1.8.9
    - wandb==0.12.11
    - webencodings==0.5.1
    - webob==1.8.7
    - websocket-client==1.3.1
    - werkzeug==2.0.3
    - wget==3.2
    - wheel==0.37.1
    - widgetsnbextension==3.5.2
    - wrapt==1.13.3
    - wtforms==3.0.1
    - wtforms-recaptcha==0.3.2
    - xlrd==2.0.1
    - xlsxwriter==3.0.3
    - yacs==0.1.8
    - yapf==0.32.0
    - yaspin==2.1.0
    - zipp==3.7.0
    - zope-deprecation==4.4.0
    - zope-interface==5.4.0
    - zope-sqlalchemy==1.6
    - zstandard==0.17.0

Additional Information

I also tried installing some of the packages using pip, and while they seem to install correctly, I receive errors indicating they are not installed when I try to use them in my project.

Request for Help

  • Could you provide guidance on resolving the missing packages and conflicting dependencies errors?
  • Are there any known compatibility issues with these packages on macOS ARM or Ubuntu?
  • Any recommended steps to ensure all packages are correctly installed and recognized by the environment?

Thank you for your assistance.

ys-zong commented

Hi, thanks for your interest. I lost access to the cluster that I used to run the experiments so maybe it's due to some hardware issues.
Can you try to set up the environment using the following commands via pip to see if it works? This should contain a minimal set of packages that you should be able to run most of the experiments.

conda create -n fair_benchmark python==3.10 -y
conda activate fair_benchmark
pip install -r requirements.txt

Inside requirements.txt:

torch
torchvision
pandas
wandb
scikit-learn
h5py
opencv-python
OApackage
Orange3
matplotlib
natsort==8.1.0
bayesian-torch==0.2.1
ys-zong commented

Closing for now due to inactivity. Feel free to reopen if the issue persists.