Paulescu/hands-on-train-and-deploy-ml

`make init` not liking lightgbm version.

bhandari-nitin opened this issue · 3 comments

Doing a make init gives me this error.

$ make init 
curl -sSL https://install.python-poetry.org | python3 -
Retrieving Poetry metadata

The latest version (1.7.1) is already installed.
poetry install
Updating dependencies
Resolving dependencies... (2.9s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing lightgbm (3.3.5): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  INFO:root:running bdist_wheel
  INFO:root:running build
  INFO:root:running build_py
  INFO:root:creating build
  INFO:root:creating build/lib
  INFO:root:creating build/lib/lightgbm
  INFO:root:copying lightgbm/callback.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/compat.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/plotting.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/__init__.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/engine.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/dask.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/basic.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/libpath.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/sklearn.py -> build/lib/lightgbm
  INFO:root:running egg_info
  INFO:root:writing lightgbm.egg-info/PKG-INFO
  INFO:root:writing dependency_links to lightgbm.egg-info/dependency_links.txt
  INFO:root:writing requirements to lightgbm.egg-info/requires.txt
  INFO:root:writing top-level names to lightgbm.egg-info/top_level.txt
  INFO:root:reading manifest file 'lightgbm.egg-info/SOURCES.txt'
  INFO:root:reading manifest template 'MANIFEST.in'
  WARNING:root:no previously-included directories found matching 'build'
  WARNING:root:warning: no files found matching '*.so' under directory 'lightgbm'
  WARNING:root:warning: no files found matching '*.so' under directory 'compile'
  WARNING:root:warning: no files found matching '*.dll' under directory 'compile/Release'
  WARNING:root:warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
  WARNING:root:warning: no previously-included files matching '*.py[co]' found anywhere in distribution
  WARNING:root:warning: no previously-included files found matching 'compile/external_libs/compute/.git'
  INFO:root:adding license file 'LICENSE'
  INFO:root:writing manifest file 'lightgbm.egg-info/SOURCES.txt'
  INFO:root:copying lightgbm/VERSION.txt -> build/lib/lightgbm
  INFO:wheel:installing to build/bdist.macosx-11.0-arm64/wheel
  INFO:root:running install
  INFO:LightGBM:Starting to compile the library.
  INFO:LightGBM:Starting to compile with CMake.
  Traceback (most recent call last):
    File "<string>", line 95, in silent_call
    File "/Users/nbhandari/mambaforge/envs/manother_ml_project_env/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['make', '_lightgbm', '-I/private/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpef06d3aq/lightgbm-3.3.5/build_cpp', '-j4']' returned non-zero exit status 2.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/Users/nbhandari/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/nbhandari/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/nbhandari/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 404, in build_wheel
      return self._build_with_temp_dir(
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
      self.run_setup()
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 480, in run_setup
      super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 334, in <module>
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 403, in run
      self.run_command("install")
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "<string>", line 248, in run
    File "<string>", line 199, in compile_cpp
    File "<string>", line 99, in silent_call
  Exception: An error has occurred while building lightgbm library file
  The full version of error log was saved into /Users/nbhandari/LightGBM_compilation.log
  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with lightgbm (3.3.5) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "lightgbm (==3.3.5)"'.

make: *** [init] Error 1

Using mamba as virtual environment. Here are the details:

$ mamba info                            

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (1.4.2) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


     active environment : manother_ml_project_env
    active env location : /Users/nbhandari/mambaforge/envs/manother_ml_project_env
            shell level : 1
       user config file : /Users/nbhandari/.condarc
 populated config files : /Users/nbhandari/mambaforge/.condarc
                          /Users/nbhandari/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=arm64
                          __osx=14.1.2=0
                          __unix=0=0
       base environment : /Users/nbhandari/mambaforge  (writable)
      conda av data dir : /Users/nbhandari/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/nbhandari/mambaforge/pkgs
                          /Users/nbhandari/.conda/pkgs
       envs directories : /Users/nbhandari/mambaforge/envs
                          /Users/nbhandari/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Darwin/23.1.0 OSX/14.1.2
                UID:GID : 502:20
             netrc file : /Users/nbhandari/.netrc
           offline mode : False

Might be because of Apple chip?

$ uname -m
arm64

nvm, it worked, had to do a brew install libomp

Also, make train wasn't working because of version of urllib3 that got updated while installing requests. so did a poetry add 'urllib3<2' and it worked.