Updating a simple package from numpy.distutils
Opened this issue · 22 comments
Hi all, I am looking for a bit of guidance in moving a simple package from numpy.distutils to scikit-build-core, following the example here.
My repo is here: https://github.com/CQCL/pyscf-ac0/tree/trying_scikit_build_core
I have a slightly more complex project structure than the example, in that I have a couple of python files interacting with a fortran90 lib (src/pyscf/cas_ac0/accas_lib.f90), I have a project executable (rdm_ac0), and it is intended to be an extension to pyscf, so the code is nested a few directories deep. I'm also a cmake novice, so I'm having trouble bringing it all together.
Ultimately, after pip install . and running the project executable rdm_ac0, I get:
Traceback (most recent call last):
File "/Users/newuser/.pyenv/versions/3.11.5/envs/py-inquanto-3.11/bin/rdm_ac0", line 5, in <module>
from pyscf.cas_ac0._cli import rdm_ac0
ModuleNotFoundError: No module named 'pyscf.cas_ac0'
similarly, if I open a python kernel and try to import, I get the same error
>>> import pyscf.cas_ac0.accas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyscf.cas_ac0'
Any help is appreciated, thanks all
The most important debugging command, IMO: unzip -l dist/*.whl (after running pipx run build[uv] --installer=uv). You'll see:
Archive: dist/pyscf_ac0-0.0.1.post1.dev9+g2e629e138c174d.d20240605-cp312-cp312-macosx_14_0_x86_64.whl
Length Date Time Name
--------- ---------- ----- ----
41436 06-05-2024 15:06 pyscf_ac0-0.0.1.post1.dev9+g2e629e138c174d.d20240605.dist-info/METADATA
114 06-05-2024 15:06 pyscf_ac0-0.0.1.post1.dev9+g2e629e138c174d.d20240605.dist-info/WHEEL
56 06-05-2024 15:06 pyscf_ac0-0.0.1.post1.dev9+g2e629e138c174d.d20240605.dist-info/entry_points.txt
35149 06-05-2024 15:06 pyscf_ac0-0.0.1.post1.dev9+g2e629e138c174d.d20240605.dist-info/licenses/LICENSE
596 06-05-2024 15:06 pyscf_ac0-0.0.1.post1.dev9+g2e629e138c174d.d20240605.dist-info/RECORD
--------- -------
77351 5 files
Nothing has been installed into the wheel. Two fixes: first, for the auto-copy scikit-build-core can do for Python files, you should either match the project name and the package name, or specify the package name explicitly (just like you would if using hatchling). So adding this:
[tool.scikit-build]
wheel.exclude = ["*.f90"]
wheel.packages = ["src/pyscf"]You'll have the following new lines:
55 06-05-2024 15:14 pyscf/cas_ac0/__init__.py
581 06-05-2024 15:14 pyscf/cas_ac0/_cli.py
485 06-05-2024 15:14 pyscf/cas_ac0/_version.py
4699 06-05-2024 15:14 pyscf/cas_ac0/accas.py
Okay, now for the built part, you need to install to the output target dir in the wheel, not the source dir.
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -8,7 +8,7 @@ find_package(
# F2PY headers
execute_process(
- COMMAND "${PYTHON_EXECUTABLE}" -c
+ COMMAND "${Python_EXECUTABLE}" -c
"import numpy.f2py; print(numpy.f2py.get_include())"
OUTPUT_VARIABLE F2PY_INCLUDE_DIR
OUTPUT_STRIP_TRAILING_WHITESPACE)
@@ -29,4 +29,4 @@ python_add_library(ac0_lib MODULE "${CMAKE_CURRENT_BINARY_DIR}/ac0_libmodule.c"
"${CMAKE_CURRENT_SOURCE_DIR}/src/pyscf/cas_ac0/accas_lib.f90" WITH_SOABI)
target_link_libraries(ac0_lib PRIVATE fortranobject)
-install(TARGETS ac0_lib DESTINATION "${CMAKE_CURRENT_SOURCE_DIR}/src/pyscf/cas_ac0")
+install(TARGETS ac0_lib DESTINATION pyscf/cas_ac0)(The PYTHON -> Python fix isn't required, as scikit-build-core does set both, but Python is the correct one for FindPython, which you are using. If you wanted to set this up to run directly without scikit-build-core, such as for an IDE, you'd need the correct one)
That adds this line:
138952 06-05-2024 15:16 pyscf/cas_ac0/ac0_lib.cpython-312-darwin.so
Don't worry, editable installs are still supported. AFAIU, that's what you want?
Thanks Henry! This is almost working. Install goes well, and the import paths resolve correctly (no more ModuleNotFoundErrors), but now I'm getting hit with a
ImportError: dlopen(/Users/newuser/.pyenv/versions/3.11.5/envs/py-inquanto-3.11/lib/python3.11/site-packages/pyscf/cas_ac0/ac0_lib.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_f2pyinitaccas_lib_'
when I try to import from pyscf.cas_ac0.ac0_lib import *. I understand this to be some kind of dynamic lib linking error, but it's to the library I've just built (I assume, given the namespace is _f2pyinitaccas_lib_), so I'm not sure how to address this.
Also, editable install were not required, but it's a nice bonus!
Weird that it is a .so on a mac, but that shouldn't matter here. It seems that the usual Reading more on the implementation, it is indeed weird that the symbol is not found there.RPATH/DYLD_LIBRARY_PATH issue is happening here. You could try to use the static libraries of f2py or add the path to its dynamic library to DYLD_LIBRARY_PATH. This is not a proper long-term solution, but first let's confirm that that is the issue here.
Not sure why this is not more common with linking to numpy though.
Thanks for the input @LecrisUT. This may muddy the waters a bit, but I am trying installation + tests on a PR here to see how it fairs on different machines (only linux and macos are of concern here).
On ubuntu-latest installation succeeds, and I replicate the error (after linking to lapack).
On macos-latest installation fails altogether. I am using gcc13 in both cases via setup-fortran.
Pretty sure there is a missing link to a main lib produced or stored by NumPy.
To start, I'm using quay.io/pypa/manylinux_2_28_x86_64. I wasn't able to use quay.io/pypa/musllinux_1_2_x86_64 as H5py doesn't seem to support it properly (and doesn't provide wheels). Lapack is required, so I had to do dnf update && dnf install lapack-devel. I'm building with pip install -v . so that I can see the CMake output. I combined a few things just to match https://numpy.org/doc/stable/f2py/buildtools/cmake.html a bit better, then found that ac0_lib-f2pywrappers2.f90 is missing; that's where this symbol comes from. If you add that to the python_add_library command, it works.
macOS is probably tricker, I think the issue there is it's mixing toolchains. In general, macOS Fortran is tricky due to llvm being the native compiler there. Will try to look into that later.
I combined a few things just to match https://numpy.org/doc/stable/f2py/buildtools/cmake.html a bit better, then found that
ac0_lib-f2pywrappers2.f90is missing
Why is it not breaking on the linking stage, and why is thr example working 🤔
It’s not breaking on the linking stage because you always leave undefined symbols enabled when compiling an extension module. But why the example is working with this missing, that I don’t know.
Ahh, this file is now always generated since 1.22.4:
From NumPy 1.22.4 onwards, f2py will deterministically generate wrapper files based on the input file Fortran standard (F77 or greater).
From https://numpy.org/doc/stable/f2py/buildtools/meson.html#automating-wrapper-generation
But I'm guessing our simple example doesn't need it, so it's basically empty (and before 1.22.4, wasn't generated at all).
Ah, maybe it's the usage of module:
from pyscf.cas_ac0 import accasGives error
ImportError: dlopen(/Users/newuser/.pyenv/versions/3.11.5/envs/py-inquanto-3.11/lib/python3.11/site-packages/pyscf/cas_ac0/ac0_lib.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_f2pyinitaccas_lib_'
Otherwise the current example has no functionality that would call #include <python.h>
Adding
"${CMAKE_CURRENT_BINARY_DIR}/ac0_lib-f2pywrappers2.f90"to the python_add_library command has indeed done the job on both my macos machine and the ubuntu runner. I can now run
from pyscf.cas_ac0 import accaslocally without any issues, and the linux workflow runs successfully.
As expected, no changes on the macos runner though.
I've managed to make some progress on this by being selective about compiler. Here's my successful attempt: https://github.com/CQCL/pyscf-ac0/actions/runs/9665312105/job/26662083580
As before, the ubuntu installation is happy, and works with gcc 11.4.0 for both the C and Fortran compilers.
For macos, I use the default clang on the runner: AppleClang 15.0.0.15000040, as the C compiler, and gcc 13.3.0 as the fortran compiler. This combination appears to work, resolving the installation errors I saw here, when I was using gcc 13 for both C and Fortran. Though I need to do a bit more testing in the coming days by building wheels for distribution.
I have been experimenting with building wheels for distribution using cibuildwheel on this PR.
I've gotten over most of the hurdles, except when I install one of my macos wheels locally and try to import, I get:
(test_3_11) ➜ dist git:(building-wheels) ✗ python -c "from pyscf.cas_ac0 import *"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/newuser/.pyenv/versions/test_3_11/lib/python3.11/site-packages/pyscf/cas_ac0/__init__.py", line 2, in <module>
from pyscf.cas_ac0 import accas
File "/Users/newuser/.pyenv/versions/test_3_11/lib/python3.11/site-packages/pyscf/cas_ac0/accas.py", line 13, in <module>
from pyscf.cas_ac0.ac0_lib import accas_lib as ac0
ImportError: dlopen(/Users/newuser/.pyenv/versions/test_3_11/lib/python3.11/site-packages/pyscf/cas_ac0/ac0_lib.cpython-311-darwin.so, 0x0002): Library not loaded: /opt/homebrew/opt/gcc@13/lib/gcc/13/libgfortran.5.dylib
Referenced from: <C867392D-407C-3F94-8867-4CD351E15810> /Users/newuser/.pyenv/versions/3.11.5/envs/test_3_11/lib/python3.11/site-packages/pyscf/cas_ac0/ac0_lib.cpython-311-darwin.so
Reason: tried: '/opt/homebrew/opt/gcc@13/lib/gcc/13/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/gcc@13/lib/gcc/13/libgfortran.5.dylib' (no such file), '/opt/homebrew/opt/gcc@13/lib/gcc/13/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)
I can resolve this with brew install gcc@13, but this feels wrong. Looking at the wheels more closely I see my linux wheels contain a collection of libraries pyscf_ac0.libs:
pyscf_ac0-0.0.post1.dev1+gfb933b2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64
├── pyscf
│ └── cas_ac0
│ ├── __init__.py
│ ├── _cli.py
│ ├── _version.py
│ ├── ac0_lib.cpython-311-x86_64-linux-gnu.so
│ └── accas.py
├── pyscf_ac0-0.0.post1.dev1+gfb933b2.dist-info
│ ├── METADATA
│ ├── RECORD
│ ├── WHEEL
│ ├── entry_points.txt
│ └── licenses
│ └── LICENSE
└── pyscf_ac0.libs
├── libblas-357956a1.so.3.4.2
├── libgfortran-040039e1.so.5.0.0
├── libgfortran-91cc3cb1.so.3.0.0
├── liblapack-1ad85175.so.3.4.2
└── libquadmath-96973f99.so.0.0.0
while the macos wheels are missing these:
pyscf_ac0-0.0.post1.dev1+gfb933b2-cp311-cp311-macosx_11_0_arm64
├── pyscf
│ └── cas_ac0
│ ├── __init__.py
│ ├── _cli.py
│ ├── _version.py
│ ├── ac0_lib.cpython-311-darwin.so
│ └── accas.py
└── pyscf_ac0-0.0.post1.dev1+gfb933b2.dist-info
├── METADATA
├── RECORD
├── WHEEL
├── entry_points.txt
└── licenses
└── LICENSE
Is this a mistake I've made at the cibw level, or in my build system?
This is what the macOS wheel repairing does - it seems you are disabling it with CIBW_REPAIR_WHEEL_COMMAND_MACOS: ""?
Aha, that was a relic from an old cibw workflow I copied over which I never questioned... but removing it seems to solve the problem! I can now install the wheel and import without issue 🎉
One remaining thing I would like to ask: for the macos wheel the pyscf_ac0.libs folder does not contain libblas or liblapack:
pyscf_ac0-0.0.post1.dev1+g9057ddc-cp311-cp311-macosx_11_0_arm64
├── pyscf
│ └── cas_ac0
│ ├── __init__.py
│ ├── _cli.py
│ ├── _version.py
│ ├── ac0_lib.cpython-311-darwin.so
│ └── accas.py
├── pyscf_ac0-0.0.post1.dev1+g9057ddc.dist-info
│ ├── METADATA
│ ├── RECORD
│ ├── WHEEL
│ ├── entry_points.txt
│ └── licenses
│ └── LICENSE
└── pyscf_ac0.dylibs
├── libgcc_s.1.1.dylib
├── libgfortran.5.dylib
└── libquadmath.0.dylib
where the linux wheel did (see above). Why is this? And is it anything to be concerned about?
Is it using accelerate, perhaps? Does it work and is it reasonably performant?
Tests run as expected and are comparable in speed (maybe even a little faster) to the previous macos wheel (the one without the libs dir). So, yes
As for whether it's using accelerate, I don't know. I'm not sure what that is. If you're referring to this, then probably not. I'm testing in a completely fresh environment
hmm, now I'm definitely not sure. I don't know how I would confirm whether it's being used. I've never actively installed accelerate myself
I've got a start here: https://github.com/scikit-build/f2py-cmake. I don't have a way to generate sig files first, like in yours (started from scratch). Feel free to advise, and maybe if you could suggest a more complex example that would illustrate building it in parts. I don't know much Fortran. :)
Neither do I to be honest 😅 @mkrompiec is the brains behind the fortran part, I've just taken on the challenge of trying to turn it into a modern looking python package and get it on pypi. I'll keep an I eye that new repo though.
As far as this issue is concerned, my project is building and wheels seem to be working, so I am happy for it to be closed 😄 Thanks again for all of your help, I've learned a lot!
I just hacked, refactored and wrapped portions of GammCor. All I’ve done is a half-automatic translation from F77 to F90, pruning unused pieces of code and making it work with f2py.