Python path error when using libpython310 on macOS

Question

Python path error when using libpython310 on macOS

speth opened this issue a year ago · 0 comments

Conda-forge documentation

I could not solve my problem using the conda-forge documentation.

Installed packages

# packages in environment at /Users/runner/miniforge3/conda-bld/cantera-recipe_1692713109591/_build_env:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h0d85af4_4    conda-forge
ca-certificates           2023.7.22            h8857fd0_0    conda-forge
cctools_osx-64            973.0.1             habff3f6_14    conda-forge
clang                     15.0.7               h694c41f_3    conda-forge
clang-15                  15.0.7          default_hdb78580_3    conda-forge
clang_osx-64              15.0.7               h03d6864_3    conda-forge
clangxx                   15.0.7          default_hdb78580_3    conda-forge
clangxx_osx-64            15.0.7               h2133e9c_3    conda-forge
compiler-rt               15.0.7               he1888fc_1    conda-forge
compiler-rt_osx-64        15.0.7               he1888fc_1    conda-forge
icu                       72.1                 h7336db1_0    conda-forge
ld64_osx-64               609                 h0fd476b_14    conda-forge
libclang-cpp15            15.0.7          default_hdb78580_3    conda-forge
libcxx                    16.0.6               hd57cbcb_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libiconv                  1.17                 hac89ed1_0    conda-forge
libllvm15                 15.0.7               he4b1e75_3    conda-forge
libsqlite                 3.42.0               h58db7d2_0    conda-forge
libxml2                   2.11.5               hd95e348_0    conda-forge
libzlib                   1.2.13               h8a1eda9_5    conda-forge
llvm-tools                15.0.7               he4b1e75_3    conda-forge
ncurses                   6.4                  hf0c8a7f_0    conda-forge
openssl                   3.1.2                h8a1eda9_0    conda-forge
python                    3.10.12         had23ca6_0_cpython    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
sigtool                   0.1.3                h88f4db0_0    conda-forge
tapi                      1100.0.11            h9ce4665_0    conda-forge
tk                        8.6.12               h5dbffcc_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
zstd                      1.5.2                h829000d_7    conda-forge

Environment info

active environment : /Users/runner/miniforge3/conda-bld/cantera-recipe_1692713109591/_build_env
    active env location : /Users/runner/miniforge3/conda-bld/cantera-recipe_1692713109591/_build_env
            shell level : 3
       user config file : /Users/runner/.condarc
 populated config files : /Users/runner/miniforge3/.condarc
          conda version : 23.3.1
    conda-build version : 3.25.0
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __osx=11.7.8=0
                          __unix=0=0
       base environment : /Users/runner/miniforge3  (writable)
      conda av data dir : /Users/runner/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/runner/miniforge3/pkgs
                          /Users/runner/.conda/pkgs
       envs directories : /Users/runner/miniforge3/envs
                          /Users/runner/.conda/envs
               platform : osx-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Darwin/20.6.0 OSX/11.7.8
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Issue

I am running into an issue with the feedstock for a package (Cantera) that uses an embedded Python interpreter by linking to libpython3.10.dylib and starting the interpreter from the Python C API. The problem is that in the embedded interpreter, sys.path refers to paths in the base conda environment, rather than the active conda environment corresponding to the libpython3.10.dylib that is linked. This then makes it impossible to import packages that are installed (only) in the active conda environment. Specifically, the normal python binary in the build environment reports sys.path as:

['',
 '$PREFIX/lib/python310.zip',
 '$PREFIX/lib/python3.10',
 '$PREFIX/lib/python3.10/lib-dynload',
 '$PREFIX/lib/python3.10/site-packages']

while the embedded interpreter's path is:

['/Users/runner/miniforge3/lib/python310.zip',
 '/Users/runner/miniforge3/lib/python3.10',
 '/Users/runner/miniforge3/lib/python3.10/lib-dynload',
 '/Users/runner/miniforge3/lib/python3.10/site-packages']

This error occurs when:

Using a conda-forge feedstock
With macOS x64 (Intel)
With Python 3.10

Other versions of Python (3.8, 3.9, and 3.11) all work fine, as do all builds for Linux and Windows.

I have produced a minimal example of this problem in the form of a PR to the cantera-feedstock repo (see conda-forge/cantera-feedstock#28). The description of that PR provides some additional details on how I've set up the test.

I have also built a minimal embedded Python test using the conda-forge build of Python 3.10 for macOS x64 outside of the conda-forge feedstock infrastructure (see repository and CI results here) that does not exhibit this problem. However, I don't understand what difference in the environment accounts for the different behavior, and of course the end goal here is that I need this working as part of the conda-forge infrastructure for the real cantera-feedstock builds.

Apologies in advance if this isn't the right place to report this issue. Thanks in advance for any help in investigating and resolving this problem.