Difficulty Setting up RPATH with scikit-build Core

Question

Difficulty Setting up RPATH with scikit-build Core

ahnaf-tahmid-chowdhury opened this issue 4 months ago · 18 comments

ahnaf-tahmid-chowdhury commented 4 months ago

I am facing some difficulty while setting up RPATH. In the past, I was using ${CMAKE_INSTALL_PREFIX}/lib for setting up RPATH. However, since I have moved to scikit-build core, I have set this to ${SKBUILD_PLATLIB_DIR}/${install-dir}/lib, but it seems the path is set as /tmp/tmp4iut2668/wheel/platlib/${install-dir}/lib and I don't know if this is the correct way.

Below is how I previously configured RPATH:

# Setup the RPATH correctly
macro(pyne_configure_rpath)
  # use, i.e. don't skip the full RPATH for the build tree
  SET(CMAKE_SKIP_BUILD_RPATH FALSE)

  # when building, don't use the install RPATH already
  # (but later on when installing)
  SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)

  SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")

  # add the automatically determined parts of the RPATH
  # which point to directories outside the build tree to the install RPATH
  SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)

  # the RPATH to be used when installing, but only if it's not a system directory
  LIST(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES
       "${CMAKE_INSTALL_PREFIX}/lib" isSystemDir)
  IF("${isSystemDir}" STREQUAL "-1")
    SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
    GET_FILENAME_COMPONENT(cxxCompilerRoot ${CMAKE_CXX_COMPILER} DIRECTORY)
    GET_FILENAME_COMPONENT(cxxCompilerRoot ${cxxCompilerRoot} DIRECTORY)
    IF(NOT "${CMAKE_INSTALL_RPATH}" STREQUAL "${cxxCompilerRoot}")
      SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}:${cxxCompilerRoot}/lib")
    ENDIF (NOT "${CMAKE_INSTALL_RPATH}" STREQUAL "${cxxCompilerRoot}")
  ENDIF("${isSystemDir}" STREQUAL "-1")
  MESSAGE("-- CMAKE_INSTALL_RPATH: ${CMAKE_INSTALL_RPATH}")
endmacro()

Answer 1 · 2024-03-23T10:45:39.000Z

How about setting the RPATH relative to $ORIGIN. The issue here is that the installed artifacts have to be fully relocatable, i.e. if you copy the files around, it should still be executable, so hard-coding absolute path RPATH breaks this.

How you put RPATH depends a lot on how/where you wish to install all dependent libraries, do you expect to have system installed veraions of those libraries, where do you want to package and what is their standard. You have to ve careful not to hard-code most of these stuff, and instead move them to pyproject.toml when possible.

Answer 2 · 2024-03-23T11:30:24.000Z

Thank you for your prompt response. You're correct. The Python module I'm using is build python3 -m build -w, typically generates the build in the temp folder.

Let me explain what I'm trying to achieve: I have a library named libpyne.so, which is installed by CMake to the site-packages/pyne/core/lib folder. My Cython modules are stored in site-packages/pyne/. Therefore, I'm seeking a method to set the relative path for libpyne.so so that other modules can easily locate it. Currently, I have to configure LD_LIBRARY_PATH in the terminal, but this isn't a robust approach.

Is there any configuration available in pyproject.toml for this? I've set my install directory to wheel.install-dir = "pyne/core" .

Answer 3 · 2024-03-23T12:44:38.000Z

You can set INSTALL_RPATH and use ${ORIGIN} inside it to point to the appropriate paths. Iiuc the cython library is defined by add_library or python_add_library right?

Note that for windows is a bit trickier, but you can get around it by making sure the dll is in the same folder.

There are 2 more design approaches to define those libraries as static, or to load the libraries manually within the python module.

Ps. The 3 recent issues seem quite similar to each other. It would be nice if the issues can be gathered for documentation improvement

Answer 4 · 2024-03-23T23:37:35.000Z

I am happy to update the documentation. There are two ways to proceed. One is to create a document page named “Migrating from Pure CMake to Skbuild,” and the other is to add an example project in the tests folder of the repository. Please let me know which procedure I should follow.

Here are my steps:

A pyproject.toml file that contains cmake.install-dir.
A CMakeLists.txt file that includes a library pointing to install-dir, and Cython and Fortran modules to a custom location that is linked with the library.

Answer 5 · 2024-03-24T22:53:02.000Z

I am encountering difficulties linking f2py with my CPython module. I've tried two approaches, but I keep getting the same error:

Python 3.11.8 (main, Feb  7 2024, 21:52:08) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from spatial_solvers import ahot_script
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/Data/Projects/pyne/tests/spatial_solvers/ahot_script.py", line 11, in <module>
    import pyne.spatialsolver
  File "/mnt/Data/Projects/pyne/.env/lib/python3.11/site-packages/pyne/spatialsolver.py", line 11, in <module>
    import pyne.transport_spatial_methods as transport_spatial_methods
ImportError: /mnt/Data/Projects/pyne/.env/lib/python3.11/site-packages/pyne/transport_spatial_methods.cpython-311-x86_64-linux-gnu.so: undefined symbol: f2pywrapmain_

Method 1:

add_library(fortranobject OBJECT "${F2PY_INCLUDE_DIR}/fortranobject.c")
target_link_libraries(fortranobject PUBLIC Python::NumPy)
target_include_directories(fortranobject PUBLIC "${F2PY_INCLUDE_DIR}")
set_property(TARGET fortranobject PROPERTY POSITION_INDEPENDENT_CODE ON)
# Common variables
set(f2py_module_name "transport_spatial_methods")
set(fortran_src_file "${PROJECT_SOURCE_DIR}/src/transport_spatial_methods/3d/main.f90")

add_custom_command(
OUTPUT ${f2py_module_name}module.c ${f2py_module_name}-f2pywrappers.f
DEPENDS ${fortran_src_file}
VERBATIM
COMMAND "${Python_EXECUTABLE}" -m numpy.f2py
        "${fortran_src_file}" -m "${f2py_module_name}" --lower)

python_add_library(${f2py_module_name} MODULE "${CMAKE_CURRENT_BINARY_DIR}/${f2py_module_name}module.c"
                  "${fortran_src_file}" WITH_SOABI)
target_link_libraries(${f2py_module_name} PRIVATE fortranobject ${CMAKE_PROJECT_NAME})

# Configure the RPATH
set_target_properties(${f2py_module_name} PROPERTIES
  INSTALL_RPATH "$ORIGIN/core/lib:$ORIGIN/../numpy.libs:$ORIGIN/../numpy/f2py/src"
  INSTALL_RPATH_USE_LINK_PATH TRUE
)
add_dependencies(${f2py_module_name} ${fortranobject})
install(TARGETS ${f2py_module_name} DESTINATION ../.) # wheel.install-dir=pyne/core wheel.packages=pyne
install(TARGETS fortranobject LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR})

Method 2:

include_directories(${F2PY_INCLUDE_DIR})
# Common variables
set(f2py_module_name "transport_spatial_methods")
set(fortran_src_file "${PROJECT_SOURCE_DIR}/src/transport_spatial_methods/3d/main.f90")
set(f2py_module_c "${f2py_module_name}module.c")

# Generate sources
add_custom_target(
  genpyf
  DEPENDS "${fortran_src_file}"
)
add_custom_command(
  OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/${f2py_module_c}"
  COMMAND ${Python_EXECUTABLE}  -m "numpy.f2py"
          -I${PROJECT_BINARY_DIR}/src
          "${fortran_src_file}"
          -m "${f2py_module_name}"
          --lower # Important
  DEPENDS ${CMAKE_PROJECT_NAME}
)

# Set up target
python_add_library( ${f2py_module_name} MODULE WITH_SOABI
  "${CMAKE_CURRENT_BINARY_DIR}/${f2py_module_c}" # Generated
  "${F2PY_INCLUDE_DIR}/fortranobject.c" # From NumPy
  "${fortran_src_file}" # Fortran source(s)
)

# Depend on sources
target_include_directories(${f2py_module_name} PUBLIC ${F2PY_INCLUDE_DIR})
target_link_libraries(${f2py_module_name} PUBLIC Python::NumPy ${LAPACK_LIBRARIES} ${BLAS_LIBRARIES} ${CMAKE_PROJECT_NAME})

# Configure the RPATH
set_target_properties(${f2py_module_name} PROPERTIES
  INSTALL_RPATH "$ORIGIN/core/lib:$ORIGIN/../numpy.libs"
  INSTALL_RPATH_USE_LINK_PATH TRUE
)
add_dependencies(${f2py_module_name} genpyf)
install(TARGETS ${f2py_module_name} DESTINATION ../.)

Answer 6 · 2024-03-25T05:01:01.000Z

Can you run readelf -d on the transport_spatial_methods.cpython-311-x86_64-linux-gnu.so. The other issue I see is with the installation paths of the targets, but I'm not sure in which library that symbol is defined. Curious when you activate a venv, does it alter the LD_LIBRARY_PATH?

add an example project in the tests folder of the repository.

There are example projects already in the docs that are being tested. But they do not cover the intricacies of RPATH or windows library loading, so it can be improved in that aspect. I think the examples can be further split based on OBJECT, STATIC, SHARED library types for the core library.

Answer 7 · 2024-03-25T08:49:56.000Z

readelf -d transport_spatial_methods.cpython-311-x86_64-linux-gnu.so

Dynamic section at offset 0x12a48 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpyne.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgfortran.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/core/lib:$ORIGIN/../numpy.libs:$ORIGIN/../numpy/f2py/src]
 0x000000000000000c (INIT)               0x5000
 0x000000000000000d (FINI)               0xddd0
 0x0000000000000019 (INIT_ARRAY)         0x13a38
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x13a40
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x260
 0x0000000000000005 (STRTAB)             0x1748
 0x0000000000000006 (SYMTAB)             0x308
 0x000000000000000a (STRSZ)              3949 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x13fe8
 0x0000000000000002 (PLTRELSZ)           2160 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x3810
 0x0000000000000007 (RELA)               0x28c8
 0x0000000000000008 (RELASZ)             3912 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x2868
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x26b6
 0x000000006ffffff9 (RELACOUNT)          48
 0x0000000000000000 (NULL)               0x0

readelf -d libpyne.so                                               

Dynamic section at offset 0x3685468 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libhdf5_serial.so.103]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libblas.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [liblapack.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libgfortran.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libpyne.so]
 0x000000000000000c (INIT)               0x45000
 0x000000000000000d (FINI)               0x319da88
 0x0000000000000019 (INIT_ARRAY)         0x3685ea8
 0x000000000000001b (INIT_ARRAYSZ)       136 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3685f30
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x260
 0x0000000000000005 (STRTAB)             0xea98
 0x0000000000000006 (SYMTAB)             0x3500
 0x000000000000000a (STRSZ)              97356 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x3686fe8
 0x0000000000000002 (PLTRELSZ)           24432 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x3ec90
 0x0000000000000007 (RELA)               0x27848
 0x0000000000000008 (RELASZ)             95304 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x27608
 0x000000006fffffff (VERNEEDNUM)         6
 0x000000006ffffff0 (VERSYM)             0x266e4
 0x000000006ffffff9 (RELACOUNT)          3510
 0x0000000000000000 (NULL)               0x0

I also tried to link Python:NumPy, F2PY_INCLUDE_DIR with libpyne.so but noticed it shows f2py inking issues.

Curious when you activate a venv, does it alter the LD_LIBRARY_PATH?

I think no, because I had previously defined the LD_LIBRARY_PATH for the PyNE library in my shell. As a result, I was unable to identify this issue. However, upon resetting it, I discovered that the linking is not working.

Answer 8 · 2024-03-25T10:11:23.000Z

I was worried the $ORIGIN was being expanded, but it seems to work fine. The other part that I can think of is where do the symbols f2pywrapmain_ get defined. I assume these are in the fortranobject or ${f2py_module_name}. If it were the latter you should consult with numpy.f2py to see if they install the library in a specific location.

What puzzles me is that the getting started example does not have any special handling, and you seem to be following the same approach in the first method. In the example there is no external library to link to, so it is more likely to work, but if f2pywrapmain_ is something defined from numpy.f2py, the same issue should happen there as well 🤔

Can you run readelf -d in the build directory as well. Normally this linking issue would occur at build time as well if you didn't have appropriate link dependencies, but that did not seem to happen right?

Answer 9 · 2024-03-25T10:43:18.000Z

Path : ~/temp/scikit-build/pyne/

readelf -d transport_spatial_methods.cpython-311-x86_64-linux-gnu.so 

Dynamic section at offset 0x12a48 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpyne.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgfortran.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [/home/tahmid/temp/scikit-build/src:]
 0x000000000000000c (INIT)               0x5000
 0x000000000000000d (FINI)               0xddc4
 0x0000000000000019 (INIT_ARRAY)         0x13a38
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x13a40
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x260
 0x0000000000000005 (STRTAB)             0x1748
 0x0000000000000006 (SYMTAB)             0x308
 0x000000000000000a (STRSZ)              3920 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x13fe8
 0x0000000000000002 (PLTRELSZ)           2160 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x37f0
 0x0000000000000007 (RELA)               0x28a8
 0x0000000000000008 (RELASZ)             3912 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x2848
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x2698
 0x000000006ffffff9 (RELACOUNT)          48
 0x0000000000000000 (NULL)               0x0

Path: ~/temp/scikit-build/src

readelf -d libpyne.so                                               

Dynamic section at offset 0x3685468 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libhdf5_serial.so.103]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libblas.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [liblapack.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libgfortran.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libpyne.so]
 0x000000000000001d (RUNPATH)            Library runpath: [/usr/lib/x86_64-linux-gnu/hdf5/serial:]
 0x000000000000000c (INIT)               0x45000
 0x000000000000000d (FINI)               0x319da88
 0x0000000000000019 (INIT_ARRAY)         0x3685ea8
 0x000000000000001b (INIT_ARRAYSZ)       136 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3685f30
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x260
 0x0000000000000005 (STRTAB)             0xea98
 0x0000000000000006 (SYMTAB)             0x3500
 0x000000000000000a (STRSZ)              97356 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x3686fe8
 0x0000000000000002 (PLTRELSZ)           24432 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x3ec90
 0x0000000000000007 (RELA)               0x27848
 0x0000000000000008 (RELASZ)             95304 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x27608
 0x000000006fffffff (VERNEEDNUM)         6
 0x000000006ffffff0 (VERSYM)             0x266e4
 0x000000006ffffff9 (RELACOUNT)          3510
 0x0000000000000000 (NULL)               0x0

Answer 10 · 2024-03-25T10:59:01.000Z

My setup

Initial setup at project/:

# Find Python and NumPy
find_package(
  Python 
  COMPONENTS Interpreter Development.Module NumPy 
  REQUIRED
)
# Check if spatial solver is requested by user
if (BUILD_SPATIAL_SOLVER)
  # Find f2py, if building spatial solver
  # Grab the variables from a local Python installation F2PY headers
  execute_process(
    COMMAND "${Python_EXECUTABLE}" -c
            "import numpy.f2py; print(numpy.f2py.get_include())"
    OUTPUT_VARIABLE F2PY_INCLUDE_DIR
    OUTPUT_STRIP_TRAILING_WHITESPACE
  )

  # Try to find LAPACK and BLAS
  find_package(LAPACK)
endif()

Main Library at project/src:

set(TRANSPORT_SPATIAL_METHODS_SRCS
  "transport_spatial_methods/3d/ahotn_kernel_module.f90"
  "transport_spatial_methods/3d/angle.f90"
  "transport_spatial_methods/3d/check.f90"
  "transport_spatial_methods/3d/dgfem_kernel.f90"
  "transport_spatial_methods/3d/echo.f90"
  "transport_spatial_methods/3d/geompack.f90"
  "transport_spatial_methods/3d/igeompack.f90"
  "transport_spatial_methods/3d/inner.f90"
  "transport_spatial_methods/3d/invar.f90"
  "transport_spatial_methods/3d/output.f90"
  "transport_spatial_methods/3d/output_phi.f90"
  "transport_spatial_methods/3d/p.f90"
  "transport_spatial_methods/3d/precision_module.f90"
  "transport_spatial_methods/3d/read_inflow_ahotn.f90"
  "transport_spatial_methods/3d/read_inflow_dgfem.f90"
  "transport_spatial_methods/3d/read_inflow_sct_step.f90"
  "transport_spatial_methods/3d/readsrc.f90"
  "transport_spatial_methods/3d/readxs.f90"
  "transport_spatial_methods/3d/sct_module.f90"
  "transport_spatial_methods/3d/sct_step_kernel_module.f90"
  "transport_spatial_methods/3d/solvar.f90"
  "transport_spatial_methods/3d/solve.f90"
  "transport_spatial_methods/3d/sweep_ahotn_l.f90"
  "transport_spatial_methods/3d/sweep_ahotn_nefd.f90"
  "transport_spatial_methods/3d/sweep_dgfem.f90"
  "transport_spatial_methods/3d/sweep_sct_step.f90"
  "transport_spatial_methods/3d/timevar.f90"
  "transport_spatial_methods/3d/trackroutines.f90"
  "transport_spatial_methods/3d/trackstruct.f90"
  "transport_spatial_methods/3d/version.f90"
  #"transport_spatial_methods/3d/main.f90"
  )

if(BUILD_SPATIAL_SOLVER)
  set(PYNE_SRCS "${PYNE_SRCS}" "${TRANSPORT_SPATIAL_METHODS_SRCS}")
endif()

add_library(${CMAKE_PROJECT_NAME} ${PYNE_SRCS})

if(BUILD_SPATIAL_SOLVER)
    target_link_libraries(${CMAKE_PROJECT_NAME} 
        PRIVATE ${LIBS_HDF5} ${BLAS_LIBRARIES} ${LAPACK_LIBRARIES}
        )
endif()

Fortarn module at project/pyne:

if(BUILD_SPATIAL_SOLVER)
  include_directories(${F2PY_INCLUDE_DIR})
  # Common variables
  set(f2py_module_name "transport_spatial_methods")
  set(fortran_src_file "${PROJECT_SOURCE_DIR}/src/transport_spatial_methods/3d/main.f90")
  set(f2py_module_c "${f2py_module_name}module.c")

  # Generate sources
  add_custom_target(
    genpyf
    DEPENDS "${fortran_src_file}"
  )
  add_custom_command(
    OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/${f2py_module_c}"
    COMMAND ${Python_EXECUTABLE}  -m "numpy.f2py"
            -I${PROJECT_BINARY_DIR}/src
            "${fortran_src_file}"
            -m "${f2py_module_name}"
            --lower # Important
    DEPENDS ${CMAKE_PROJECT_NAME}
  )

  # Set up target
  python_add_library( ${f2py_module_name} MODULE WITH_SOABI
    "${CMAKE_CURRENT_BINARY_DIR}/${f2py_module_c}" # Generated
    "${F2PY_INCLUDE_DIR}/fortranobject.c" # From NumPy
    "${fortran_src_file}" # Fortran source(s)
  )

  # Depend on sources
  target_include_directories(${f2py_module_name} PUBLIC ${F2PY_INCLUDE_DIR})
  target_link_libraries(${f2py_module_name} PUBLIC Python::NumPy ${LAPACK_LIBRARIES} ${BLAS_LIBRARIES} ${CMAKE_PROJECT_NAME})
  
  # Configure the RPATH
  set_target_properties(${f2py_module_name} PROPERTIES
    INSTALL_RPATH "$ORIGIN/core/lib"
  )
  add_dependencies(${f2py_module_name} genpyf)
  install(TARGETS ${f2py_module_name} DESTINATION ../.)
endif()

Answer 11 · 2024-03-25T11:36:51.000Z

Nitpick: don't write stuff like *_SRCS, these are relics of autotools. Use target_source and such. Also avoid using ${PROJECT_NAME} etc. for simple naming of targets, libraries, etc. it makes it hard to navigate.

The only things I notice are:

libpyne.so does not have an equivalent RPATH, but it seems to point to a system library so it should be fine without it.
The only RPATH for transport_spatial_methods.cpython-311-x86_64-linux-gnu.so seems to point to libpyne.so. But I also don't see the install directive for pyne target to check if it's installed in core/lib or other paths that you've added there.

Another thing you can do is to try and build it outside of SKBUILD, and with some PYTHONPATH manipulation (point it to the root of package/__init__.py, the built python_add_library etc., maybe check this snippet) and try to run these in the build directory. If that fails, the issue is not on the install side.

Answer 12 · 2024-03-30T18:55:49.000Z

You are right. The issue was occurring due to main.f, not from the installation side, and it is now resolved.

However, before closing this issue, I have a question to ask: is there any variable available from the scikit-build side that performs a similar function to the one below?

set(SKBUILD_LIB_DIR ${CMAKE_INSTALL_LIBDIR}/python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}/site-packages/${install-dir}/${CMAKE_INSTALL_LIBDIR})

I know there is SKBUILD_PLATLIB_DIR, but I think it contains the full path. Also, I don't know if there any variable that contains ${install-dir}.

Answer 13 · 2024-03-30T18:58:35.000Z

You cannot install to site-packages directly when making a wheel. You must install to the wheel and then let pip or installer unpack it into site-packages.

Answer 14 · 2024-03-30T19:02:04.000Z

Actually, I am not planning to install directly from the CMake. I will be using this for setting up RPATH, from env/bin to install-dir/lib.

Answer 15 · 2024-03-30T19:04:40.000Z

Here is my current approach:

    set (SKBUILD_LIB_DIR ${CMAKE_INSTALL_LIBDIR}/python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}/site-packages/pymoab/core/${CMAKE_INSTALL_LIBDIR})

# Set the install RPATH
if(APPLE)
    set_target_properties(compareFiles PROPERTIES INSTALL_RPATH "@loader_path/../${SKBUILD_LIB_DIR}")
elseif(UNIX)
    set_target_properties(compareFiles PROPERTIES INSTALL_RPATH "$ORIGIN/../${SKBUILD_LIB_DIR}")
endif()

Answer 16 · 2024-03-30T19:08:55.000Z

I’ll have to look, I’m worried about making assumptions that might not always be true. FYI FindPython provides the site packages dir as Python_SITEARCH (https://cmake.org/cmake/help/latest/module/FindPython.html).

Answer 17 · 2024-03-30T19:19:19.000Z

Unfortunately, this also sets the full path.

Example:

/usr/local/lib/python3.11/dist-packages

I am looking for something like:

lib/python3.11/dist-packages

More precisely:

lib/python3.11/dist-packages/${install-dir}/lib

Answer 18 · 2024-03-31T05:44:28.000Z

I think you can get the ROOT dir and compute it with cmake's relative path computation. But I'm pretty sure this is not something you can put in a wheel. When you set up Python, you can configure these values. dist-packages is not always separate from site-packages. And if you were to ever use ABI3, the Python number could change.

In general, I think you can't make links between the data dir and the platlib dir. It's also fairly unfriendly to users to put things in data, since if they don't use virtual environments, they are getting things installed directly to /usr/local or wherever Python consider's it's root. Most packages put everything in platlib, then all paths are guaranteed to be relocatable when the wheel is installed. If you do put things in data or bin, it shouldn't depend on finding platlib unless it can query it from Python at runtime.