Pyyaml error on poetry install

Question

Pyyaml error on poetry install

Closed this issue 6 months ago · 10 comments

Problem Description

I am trying to install cleanrl by following the installation guide https://docs.cleanrl.dev/get-started/installation/ but I am running into an error when poetry tries the Installing pyyaml step. I searched for this issue in the doc, current issues and discord channel but couldn't find anything.

Checklist

I have installed dependencies via poetry install (see CleanRL's installation guideline.
I have checked that there is no similar issue in the repo.
I have checked the documentation site and found not relevant information in GitHub issues.

Current Behavior

I am working on an Ubuntu 22.04.2 LTS machine with Python 3.10.12. On cloning the repo and installing Poetry (I tried the latest version 1.6.1 and also 1.4.1 and 1.3.1, but experienced the same error with all versions) I get this error

(clearnrl) /content/cleanrl# /root/.local/bin/poetry install
Installing dependencies from lock file

Package operations: 77 installs, 1 update, 1 removal

  • Removing pip (22.3.1)
  • Installing numpy (1.21.6)
  • Installing certifi (2023.5.7)
  • Installing charset-normalizer (3.1.0)
  • Installing idna (3.4)
  • Installing markupsafe (2.1.2)
  • Installing pyasn1 (0.5.0)
  • Installing six (1.16.0)
  • Installing smmap (5.0.0)
  • Installing typing-extensions (4.5.0)
  • Installing urllib3 (1.26.15)
  • Installing absl-py (1.4.0)
  • Installing cachetools (5.3.0)
  • Installing cloudpickle (2.2.1)
  • Installing cycler (0.11.0)
  • Installing fonttools (4.38.0)
  • Installing gitdb (4.0.10)
  • Installing gym-notices (0.0.8)
  • Installing kiwisolver (1.4.4)
  • Installing oauthlib (3.2.2)
  • Installing packaging (23.1)
  • Installing pillow (9.5.0)
  • Installing pyasn1-modules (0.3.0)
  • Installing pygments (2.15.1)
  • Installing pyparsing (3.0.9)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2023.3)
  • Installing pyyaml (5.4.1): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  /tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
  !!
  
          ********************************************************************************
          The license_file parameter is deprecated, use license_files instead.
  
          By 2023-Oct-30, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.
  
          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************
  
  !!
    parsed = self.parsers.get(option_name, lambda x: x)(value)
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/root/.local/share/pypoetry/venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/root/.local/share/pypoetry/venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/root/.local/share/pypoetry/venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
      self.run_setup()
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command
      super().run_command(command)
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 318, in run
      self.find_sources()
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
      mm.run()
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 548, in run
      self.add_defaults()
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 586, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/command/sdist.py", line 113, in add_defaults
      super().add_defaults()
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "<string>", line 201, in get_source_files
    File "/tmp/tmpv0b7jscm/.venv/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:147 in _prepare
      143│ 
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│ 
      146│             if error is not None:
    → 147│                 raise error from None
      148│ 
      149│             return path
      150│ 
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Expected Behavior

A successful installation of cleanrl

Possible Solution

As a workaround, I tried installing pyyaml manually using pip without the PEP 517 build process, but that did not solve the issue

pip install --no-use-pep517 pyyaml==5.4.1

If I downgrade the version of pyyaml==5.3.1 this error goes away but there are other packages that depend on pyyaml==5.4.1 so the installation fails again

Steps to Reproduce

git clone https://github.com/vwxyzjn/cleanrl.git
curl -sSL https://install.python-poetry.org | python3
poetry install

Answer 1 · 2023-09-21T13:52:09.000Z

I managed to get around this by installing requirements.txt files, but I had to manually downgrade the version of pyyaml to 5.3.1 and also install the latest version of flax 0.7.4 because I got this error flax 0.6.4 depends on PyYAML>=5.4.1

# core dependencies
pip install -r requirements/requirements.txt

# optional dependencies
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-mujoco_py.txt
pip install -r requirements/requirements-procgen.txt
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-jax.txt
pip install -r requirements/requirements-docs.txt
pip install -r requirements/requirements-cloud.txt

Answer 2 · 2023-09-21T19:50:56.000Z

another approach that worked for me with Poetry is add pyyaml = {version = "5.3.1", optional = true} to the pyproject.toml and then changed jax = {version = "0.4.16", optional = true} and flax = {version = "0.4.2", optional = true} as they need pyyaml to 5.4.1

Answer 3 · 2023-10-12T12:27:50.000Z

Sorry this is an issue. We'd have to refactor dependency stuff in at some point. For now probably just use py3.8

Answer 4 · 2023-11-19T22:21:23.000Z

Any updates on this? I tried 3.8 and I get this error try to install jax:

ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  
  *** Orbax is a namespace, and not a standalone package. For model checkpointing and exporting utilities, please install `orbax-checkpoint` and `orbax-export` respectively (instead of `orbax`). ***

On 3.10 implementing hom-bahrani's suggestions I get this error when trying jax env ppo:

poetry run python cleanrl/ppo_atari_envpool_xla_jax.py --help
2023-11-19 17:19:53.180139: W external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-11-19 17:19:53.195599: W external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-11-19 17:19:53.196676: W external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/home/vadim/cleanrltest2/cleanrl/cleanrl/ppo_atari_envpool_xla_jax.py", line 20, in <module>
    import optax
  File "/home/vadim/.cache/pypoetry/virtualenvs/cleanrl-qeHxug-V-py3.10/lib/python3.10/site-packages/optax/__init__.py", line 17, in <module>
    from optax import contrib
  File "/home/vadim/.cache/pypoetry/virtualenvs/cleanrl-qeHxug-V-py3.10/lib/python3.10/site-packages/optax/contrib/__init__.py", line 17, in <module>
    from optax._src.contrib.mechanic import MechanicState
  File "/home/vadim/.cache/pypoetry/virtualenvs/cleanrl-qeHxug-V-py3.10/lib/python3.10/site-packages/optax/_src/contrib/mechanic.py", line 34, in <module>
    import chex
  File "/home/vadim/.cache/pypoetry/virtualenvs/cleanrl-qeHxug-V-py3.10/lib/python3.10/site-packages/chex/__init__.py", line 17, in <module>
    from chex._src.asserts import assert_axis_dimension
  File "/home/vadim/.cache/pypoetry/virtualenvs/cleanrl-qeHxug-V-py3.10/lib/python3.10/site-packages/chex/_src/asserts.py", line 26, in <module>
    from chex._src import asserts_internal as _ai
  File "/home/vadim/.cache/pypoetry/virtualenvs/cleanrl-qeHxug-V-py3.10/lib/python3.10/site-packages/chex/_src/asserts_internal.py", line 34, in <module>
    from chex._src import pytypes
  File "/home/vadim/.cache/pypoetry/virtualenvs/cleanrl-qeHxug-V-py3.10/lib/python3.10/site-packages/chex/_src/pytypes.py", line 26, in <module>
    ArrayDevice = jax.Array
AttributeError: module 'jax' has no attribute 'Array'

Answer 5 · 2023-11-19T22:30:50.000Z

Yes could you try #424

Answer 6 · 2023-11-19T22:46:52.000Z

Yes could you try #424

Thanks, will do.

Answer 7 · 2023-11-22T02:21:15.000Z

I have the same issue. Are there any new solutions?

Answer 8 · 2023-11-22T05:09:47.000Z

@flypark666 have you tried #424 ?

Answer 9 · 2023-11-23T07:10:25.000Z

Sorry, I do not know what part of #424 should I try.

Answer 10 · 2023-11-28T01:34:30.000Z

@flypark666 and @hom-bahrani and @vadimbarkhatov this is resolved in main. I added CI for 3.8, 3.9, 3.10 in #424. Please give it a try.

(cleanrl-py3.10) ➜  cleanrl git:(refactor-tyro) ✗ poetry install
Installing dependencies from lock file

Package operations: 79 installs, 2 updates, 1 removal

  • Removing pip (22.3.1)
  • Installing numpy (1.24.4)
  • Installing certifi (2023.5.7)
  • Installing charset-normalizer (3.1.0)
  • Installing idna (3.4)
...
 • Installing pyyaml (6.0.1)
...
  • Installing tenacity (8.2.3)
  • Installing tensorboard (2.11.2)
  • Installing tyro (0.5.10)

Installing the current project: cleanrl (2.0.0b1)