tomerfiliba-org/reedsolomon

ERROR: Failed building wheel for reedsolo

Franky1 opened this issue · 10 comments

Franky1 commented
  • Microsoft Windows 10 Home x64
  • Python 3.8.6

Unexpected behaviour

pip install reedsolo

fails with:

Collecting reedsolo
  Using cached reedsolo-1.5.4.tar.gz (271 kB)
Building wheels for collected packages: reedsolo
  Building wheel for reedsolo (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\WinPython\WPy64-3860\python-3.8.6.amd64\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-sknnm65z\\reedsolo_d0f8e6cce4fe4abfbb92aa3abc1928b7\\setup.py'"'"'; __file__='"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-sknnm65z\\reedsolo_d0f8e6cce4fe4abfbb92aa3abc1928b7\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\xxxxx\AppData\Local\Temp\pip-wheel-_7lq_i9f'
       cwd: C:\Users\xxxxx\AppData\Local\Temp\pip-install-sknnm65z\reedsolo_d0f8e6cce4fe4abfbb92aa3abc1928b7\
  Complete output (10 lines):
  Cython is installed, building creedsolo module
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.8
  copying reedsolo.py -> build\lib.win-amd64-3.8
  running build_ext
  building 'creedsolo' extension
  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
  ----------------------------------------
  ERROR: Failed building wheel for reedsolo
  Running setup.py clean for reedsolo
Failed to build reedsolo
Installing collected packages: reedsolo
    Running setup.py install for reedsolo ... error
    ERROR: Command errored out with exit status 1:
     command: 'C:\WinPython\WPy64-3860\python-3.8.6.amd64\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-sknnm65z\\reedsolo_d0f8e6cce4fe4abfbb92aa3abc1928b7\\setup.py'"'"'; __file__='"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-sknnm65z\\reedsolo_d0f8e6cce4fe4abfbb92aa3abc1928b7\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\xxxxx\AppData\Local\Temp\pip-record-qw0ikq9c\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\WinPython\WPy64-3860\python-3.8.6.amd64\Include\reedsolo'
         cwd: C:\Users\xxxxx\AppData\Local\Temp\pip-install-sknnm65z\reedsolo_d0f8e6cce4fe4abfbb92aa3abc1928b7\
    Complete output (10 lines):
    Cython is installed, building creedsolo module
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.8
    copying reedsolo.py -> build\lib.win-amd64-3.8
    running build_ext
    building 'creedsolo' extension
    error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\WinPython\WPy64-3860\python-3.8.6.amd64\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-sknnm65z\\reedsolo_d0f8e6cce4fe4abfbb92aa3abc1928b7\\setup.py'"'"'; __file__='"'"'C:\\Users\\xxxxx\\AppData\\Local\\Temp\\pip-install-sknnm65z\\reedsolo_d0f8e6cce4fe4abfbb92aa3abc1928b7\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\xxxxx\AppData\Local\Temp\pip-record-qw0ikq9c\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\WinPython\WPy64-3860\python-3.8.6.amd64\Include\reedsolo' Check the logs for full command output.

Expected behaviour

pip installation should run without need to install massive ms toolchain. Provide wheels?

Franky1 commented

I am using WinPython distributions for all of my windows machines:
https://winpython.github.io/
I suspect something is broken with this particular WinPython version.
I just tried it on another computer with (Win)Python 3.7.7, there pip install --upgrade reedsolo worked without errors.
I will have another look at the other machine with (Win)Python 3.8.6. and report back.

Franky1 commented

Still after switching from WinPython 3.8.6.0 to WinPython 3.8.8.0 on this machine, i had no success to install reedsolo.

Only after installing the latest Microsoft Visual C++ 14.2 Build Tools on this machine I was able to install reedsolo without any error messages.

https://wiki.python.org/moin/WindowsCompilers

I think that issue can be closed with this.

Franky1 commented

I still wondered why a ready-made wheel is not taken instead.
Could it be because only one wheel is provided for Python 3.7?

eirinnm commented

Only after installing the latest Microsoft Visual C++ 14.2 Build Tools on this machine I was able to install reedsolo without any error messages.

https://wiki.python.org/moin/WindowsCompilers

I think that issue can be closed with this.

The problem with this solution is it requires installing a 6.8GB build tool. Pip should provide a compiled binary avoiding the need for it.

edit: found a solution: pip install reedsolo --no-binary={reedsolo}

lrq3000 commented

To answer both of your questions, I GUESS (but I'm not sure) that you both had Cython installed but without a C compiler compatible with your Python version. So reedsolo tried to compile on pip install but failed miserably.

But note that there IS a compiled binary in the wheel, that's why I went through the hassle of configuring a quite complex setup.py to manage different cases. But Python is NOT a language made for compilation, so this kind of use case is a bit off limits, and errors like this can happen. Maybe now with pip3 it works better, fairly complex tools are now installable with pip instead of requiring precompiled binaries through conda or other recipe forges, but I don't have the time to rework everything.

I have merged some proposed changes to fix this issue hopefully. Please let me know if it now works (I know it's quite late, but it's all pro-bono voluntary work here, and it's not even my specialty field).

Please re-open this thread if necessary. Thank you for your patience.

lia800 commented

I still have problems... What can I do? I have Python 3.11. Windows.
image

lrq3000 commented

@lia800 Thank you for reporting this issue. Could you please try to install the development release of reedsolo like this:

pip install --upgrade git+https://github.com/tomerfiliba/reedsolomon

And please let me know if it works now? I have just pushed some changes that should hopefully fix the issue once and for all (ie, only the pure-python implementation should be installed by default now, the cythonized implementation requires an explicit switch, it's not compiled automatically anymore).

lia800 commented

Thank you very much [lrq3000], but I realized that I needed to download Microsoft C++ Build Tools and that works.

lrq3000 commented
  • 😄1
lrq3000 commented

For future reference, here is a post-mortem of this issue: the main issue is that I tried to provide a compiled creedsolo (cythonized extension) in the wheel build for Windows users, the assumption was that Linux users either did not have Cython installed, or they had Cython + GCC, as it's highly unlikely Linux users would not have GCC installed.

However, there were two pitfalls:

  • The case of having Cython without GCC could happen both on Windows and Linux, but especially on Windows, with some users installing Cython but not GCC.
  • Normally the Windows wheel should have a compiled creedsolo extension, but it only worked for the latest Python release for which I compiled the cythonized extension.

The proper way to build wheels with cythonized extensions is in fact to compile one wheel build for each Python release we support and for each platform (Linux, MacOS and Windows), that's a lot! And also for each architecture (eg, arm64 would need another set of wheels).

In practice, the technology did not exist when we started offering a cythonized extension in this project. Nowadays, there are experimental solutions, such as cibuildwheel and multibuild, both leveraging manylinux, with which I am not familiar but appears to be a containerized implementation of linux. Also, we could probably drop our current custom implementation of setup.py arguments to make cythonization optional, by using cython-setuptools instead.

Given how complex and experimental these solutions are at the moment, although very promising, I think it's more reasonable and more future proof to simply make the cythonized extension explicitly optional: by default, only the pure python version is implemented, but the cython source code is available and can be compiled at anytime, either at install time via pip install reedsolo --install-option="--cythonize", or locally with python setup.py install --cythonize.

For future reference, here are some additional pertinent resources: