Gooey installation problems
Closed this issue · 45 comments
My install of nemo fails with the setuptools problem, but even though i try pip install setupools==58, and separately try Gooey .., no joy. I am using win 11 and Python in venv.
PS C:\Users\xxx> pip install setuptools==58
Requirement already satisfied: setuptools==58 in c:\users\xxx\venv\lib\site-packages (58.0.0)
DEAP ran
Gooey fails
Command '"C:\Users\xxx\venv\Scripts\python.exe" -c "import setuptools, distutils.msvc9compiler as msvc; mc = msvc.MSVCCompiler(); mc.initialize(); print(mc.cc)"' failed with exit code 1.
PS where should nemo.cfg be placed?
Are you able to install Gooey using pip install Gooey? If that package doesn't install, you'll need to take that up with chriskiehl/Gooey.
nemo.cfg can go anywhere. You should set the environment variable NEMORC to point to the location of the file.
Please follow up on the nemo-devel mailing list.
These issues relate to the Gooey package, not NEMO. NEMO just uses the Gooey package to get its job done. Any problems have to be reported to the relevant maintainer, sorry.
Ah, sorry, I misunderstood. I will check it out, thanks.
I have added Python 3.9 and 3.10 to the list of environments to build and test with in Github workflows. That exposed another problem: nosetests is unmaintained and doesn't work on Python 3.10. I'll need to fix that, too.
I can tell you that "deap" is no longer being maintained and there is an issue with it and it will not install.
error: metadata-generation-failed
deap is using commands in it's build system that have been removed. It is using "use_2to3" which was removed about a year ago.
Because NEMO is using a requirement that is no longer maintained and has no chances of being fixed a different library needs to be located that can provide the same functionality and is still actively being maintained. The last release of deap was in 2019.
There is no easy way to monkey patch around this problem either. This is due to pip running the setup program in a sandbox type of environment.
If I downgrade setup tools to 57.5.0 deap will install properly but then we hit a road block with wxPython which is where the MSVC library I made comes into play.
One of the DEAP maintainers reassures me that DEAP is still maintained.
I just installed DEAP from source code using Python 3.10:
python3.10 setup.py build install --prefix=$HOME/tmp
I got it all going and it all compiles properly.
The problem is Gooey (I think that's what it's called) as it uses wxPython and wxPython does not compile on a Windows 10+ machine running Python 3.10
We have been working to correct that problem.
I added a pypropject.toml file that handles getting the right version of setuptools installed. that takes care of the deap issue. I have a modified version of wxPython that compiles properly and I added that to the list of dependencies at the top of the list. So setuptools grabs the wxPython from a branch of a fork I have for wxPython and it will compile and install it using that. So any other library that needs wxPython the requirement will already be satisfied.
I did as best I could as far as keeping any code alterations to NEMO as small as possible. Onle changes are the addition of the pyproject.toml file and some changes to the setup.py file.
https://github.com/kdschlosser/NEMO/tree/fixes_build_python_310
pip install -vvv --no-cache-dir https://github.com/kdschlosser/NEMO/archive/7e0f8b0a27e374b2f27edc1948f4b38ceef1e895.tar.gz
I also had to modify requirements.txt
It's all really simple things and once either setup tools gets fixed or the libraries makes changes that allow them to install properly it is easy to remove the dependencies I added.
I also added pyMSVC to pypi. I will be updating my PR wit wxPython so it grabs the repository from pypi instead of including it with the wxPython build system. I had to wait on doing that because comtypes would not run properly on Python 3.10 and now that has been fixed I was able to put pyMSVC up onto pypi.
It's a tangles mess of issues python has created by deprecating distutils and removing all of the py2to3 stuff. There is no single standard to install packages, you have pip and easyinstall coupled with setup.py, setup.cfg, requirements.txt and pyproject.toml.
There is still no defacto bundling too yet, IDK if they are going to use wheels or eggs. but right now it is a mish mosh of both of them.
The problem is they never came up with a plan. it is all still flying by the seat of the pants. If the developers of Python cam up with a plan and laid out the tracks for that plan and stick to it like glue there would be a single build ecosystem that would work.
Getting anything fixed in Python that has to deal with Windows can take a decade or more. There are issues with sockets and Windows that have been reported to Python more then a decade ago and they have not been fixed. Python is written by posix developers and they could care less about Windows users and as such bugs don't get fixed or changes get made without any regard for the impact to Windows users. It's a lot of having to work around problems until proper fixes get written and added which unfortunately can take a really long time.
I have personally been trying to get setuptools to fix they MSVC portion of the build system for several years now. and during that time I had to come up with something that would work and that is where pyMSVC came from. Now I don't even bother with setuptool or distutils and if i need to get something to compile in the build system I load that script and it works without fail.
I have never had to build wxPython, even on Linux, because PyPI pulled in a precompiled package. The main issue here is that there is no precompiled package for Python 3.10. I would suggest asking Robin Dunn to build a Python 3.10 package and put it up on PyPI. You have better things to be doing than compiling wxWidgets.
We know what the problem is and getting the folks over at setuptools to fix it doesn't seem to be happening. The build process for posix systems works fine and that is why there is a wheel for it.
This is a solution until the folks over at setuptools decide that they want to fix the compiler detection on Windows.
This whole problem is an upstream problem and no one upstream seems to want to fix it. Nothing I can do about that except provide a way to work around it so that YOUR users are able to install this library. If you are not interested in providing a solution for the people that are running Windows then you fall right into the same group as the people that maintain setuptools and Python. Considering that 76.33% of all computers in the world are running Windows not providing a solution for it's users really doesn't make a bit of sense. I am not sure why the Python developers and setuptools developers have a mentality that Windows is the minority but it's not.
If you are not interested in providing a solution then let me know. I will delete the code and move on to something different.
I ran DEAP through 2to3 and removed use_2to3 from its setup.py file. On the surface of it, that worked fine. I will encourage the DEAP maintainers to do this as a priority: DEAP/deap#658
Ben, can you confirm what will work for windows & Python 3.10 for nemopt?
I can't confirm that, sorry. I had not used Python 3.10 until two days ago. I am working on getting the issues fixed in the dependent packages, though.
@Gerryjj There is a wxPython wheel for Python 3.10 on Windows here: https://github.com/oleksis/youtube-dl-gui/releases/tag/v1.8.2. You can install that with pip, then run pip install Gooey and you should be sorted.
DEAP doesn't currently work on Python 3.10, so this is a bit of a moot point -- nor will NEMO. I am happy to help get the dependent packages in question fixed, but am not keen to use workarounds in the NEMO source code. I'd rather pin packages in requirements.txt if that's what is needed.
The only way to handle the deap issue is with the pyproject.toml. This is the only way to ensure that the proper version of setuptools gets installed for the installation that is what takes care of the issue with deap.
it is moot to add a dependency to requiements.txt if there is a pyproject.toml file. I also do not believe that requirements.txt gets processed if someone does this
python setup.py install
the only way the requirements.txt file gets processed is when pip gets run. where as the pyproject.toml file gets processed if using pip or running the setup.py file directly.
For now, I have setup.py downloading my fork of DEAP from Github that has been converted to Python 3. I am confident this will be fixed by the DEAP maintainers soon.
OK cool. so making progress on it. There were issues that did need to be corrected. Granted it's not your code that needs to get corrected and it's the code in some of the requirements causing the issue. Sometimes you have to rattle some cages else where in order to get things all fixed up. I know it's a pain to have to do that and it really shouldn't be your responsibility to fix it either.
I hate having to band-aide things to get them to work. Sometimes that is all a person can do if the problem is upstream as with this. Everything can be made to function with a small amount of code being changed.
Oh I did want to mention. That wxPython wheel you found, that is compiled using my fork of wxPython, the same fork I used when updating your code.
DEAP 1.3.2 has been released.
Good deal!!
My own system is now running Python 3.10 (Ubuntu 22.04) and a pip install nemopt is working fine, with the exception of SCOOP. I am just prodding the SCOOP maintainer to upload a new release to PyPI.
I've never used PyCharm and cannot possibly hope to support all development environments, sorry. I've just recently had to learn VS Code to support another user.
nemopt is the name of the PyPI package. That's the only time that name is used (nemo was taken). On my Linux system, this is where the nemopt package gets installed:
/home/bje/.local/lib/python3.10/site-packages/nemo
If you want to test the installation of the prerequisite packages, this little script should be sufficient:
import numpy
import pandas
import matplotlib
import deap
import pint