Install problems
helgestein opened this issue · 6 comments
On macOS you cannot install nomad-lab>=1.2.0 because of the pymatgen depency pymatgen==2023.2.28 failing to build a wheel causing the entire install to fail. The same happens in any docker container run on the apple silicon platform.
Simple way of reproducing this error on a M1+ chip is to make a minimal python3 docker file with
RUN pip install nomad-lab
Hi @helgestein!
Could you share the install logs from your attempt to install nomad-lab=1.2.0
on Apple silicon? Have you succeeded in installing an older version of nomad-lab
or pymatgen
on Apple silicon?
It would also be interesting to see what kind of errors you get with docker when trying to run our docker image (the latests one is: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:latest
): in principle, the issue in docker should be something else, since the wheel is already built for you.
I hope we can find some solution, but it is certainly a challenge to ensure support for Apple silicon, since we can't e.g. run systematic installation tests on this specialized hardware.
Sure!
We should maybe split the docker and pip discussion simply because running the docker on most laptops will eat too much resources. See also the general comment on dependencies below.
so running pip install nomad-lab
on macOS Sonoma 14.1.1 on a M2 chipset on python 3.10.10 results in:
(truncating the output)
Error compiling Cython file:
------------------------------------------------------------
...
#ensure correct shape
fcoords1, fcoords2 = np.atleast_2d(fcoords1, fcoords2)
pbc = lattice.pbc
cdef int n_pbc = sum(pbc)
cdef int n_pbc_im = 3 ** n_pbc
^
------------------------------------------------------------
pymatgen/util/coord_cython.pyx:93:26: Cannot assign type 'double' to 'int'
Compiling pymatgen/util/coord_cython.pyx because it changed.
[1/1] Cythonizing pymatgen/util/coord_cython.pyx
Traceback (most recent call last):
File "/Users/helgesorenstein/mambaforge/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/helgesorenstein/mambaforge/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/helgesorenstein/mambaforge/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 410, in build_wheel
return self._build_with_temp_dir(
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
self.run_setup()
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 26, in <module>
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/normal/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 368, in run
self.run_command("build")
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 89, in run
_build_ext.run(self)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
self.build_extensions()
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
self._build_extensions_serial()
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
self.build_extension(ext)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 250, in build_extension
_build_ext.build_extension(self, ext)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/Cython/Distutils/build_ext.py", line 130, in build_extension
new_ext = cythonize(
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
cythonize_one(*args)
File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: pymatgen/util/coord_cython.pyx
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pymatgen
Failed to build pymatgen
ERROR: Could not build wheels for pymatgen, which is required to install pyproject.toml-based projects
running pip install nomad-lab -prefer-binary
results in the same issue. Installing the latest pymatgen version (2023.5.10) via pip install pymatgen
works just fine. It really seems like that exact version of pymatgen that you have as a dependency (2023.2.28) fails.
Overall, I kind of disagree with the necessity to install all kinds of theory-focused dependencies when the goal is to write and test a parser (especially as an experimentalist) for a nomad-oasis. I'm guessing your goal is widespread adoption beyond computational chemistry.
Another issue here is that phonopy is also causing issues upon installation on windows machines as it nessecitates a VS C++ compiler. These errors upon installing make the entry barrier even higher for new adopters outside computational chemistry.
Ok so creating a conda venv with python 3.9 will at least install nomad-lab, a nomad import will work BUT trying to start to develop an EntryArchive will fail because the numpy version installed as part of nomad-lab will result in:
Python 3.9.0 | packaged by conda-forge | (default, Nov 26 2020, 07:55:15)
[Clang 11.0.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
import numpy as np
np.version
'1.22.4'
from nomad.datamodel import EntryArchive, EntryMetadata
...
raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'float128'
numpy in that version does have numpy.longdouble, which is the same as float128, but float128 is not available on macros/numpy. When installing in Python 3.9, updating numpy will not work besides numpy's latest version on macos and python 3.11.8 does not have float128 either. Besides I cannot really think of a utter nessesity to carry hat level of precision in any matsci experiment.
The code in the utils functions init does the following:
float_numpy = {np.float16, np.float32, np.float64} | (set() if os.name == 'nt' else {np.float128}
so maybe add a try except with np.longdouble?
Thanks for the additional details and comments!
-
Installing the latest pymatgen version (2023.5.10) via pip install pymatgen works just fine. It really seems like that exact version of pymatgen that you have as a dependency (2023.2.28) fails.
This might have something to do with Cython going through some breaking changes when migrating to 0.30, which has caused us some trouble previously as well. I think we can quite easily bump up the version number to see if this helps.
-
Overall, I kind of disagree with the necessity to install all kinds of theory-focused dependencies when the goal is to write and test a parser (especially as an experimentalist) for a nomad-oasis. I'm guessing your goal is widespread adoption beyond computational chemistry.
You are definitely right here. Historically NOMAD has been very theory-focused, thus the current dependencies. But we are actively migrating to a more plugin-based approach, where the idea is to keep the main package as lean as possible, and then users can bring in additional dependencies through plugins. Unfortunately, this is not yet done, but is underway. With these changes, also
phonopy
will become a part of an optional plugin. I can check if there is an intermediate solution wherephonopy
gets installed only if the user requires theparsing
extras withpip install nomad[infractructure, parsing].
-
When installing in Python 3.9, updating numpy will not work besides numpy's latest version on macos and python 3.11.8 does not have float128 either. Besides I cannot really think of a utter nessesity to carry hat level of precision in any matsci experiment.
I agree that float128 should no be required by default: I don't think I have seen anyone use it, but it was included just in case. We can make it optionally with a try/catch block as you suggest.
Overall, I will see if I can address 1,3 and the phonopy
problem in a new MR. If you have then time, you could test these changes to see if your problems get resolved or if additional steps are needed.
I have now merged a proposed fix into the develop
branch.
Unfortunately, I can't test it thoroughly since I don't have the hardware available. If you have time, you could try to clone the develop
branch and install the nomad-lab package from there with pip install .
This is not exactly the same as installing our PyPI package, but would at least indicate if some of the problems are gone. There is also an updated docker image at: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:develop
.
Closing for now.