cffi not working under a virtual environment in Mac

[scarlehoff]

The gist of the problem is that, as soon as the python interpreter reaches the "cffi code", it loses knowledge about the virtual environment. Reading the documentation I guess this is somewhat expected (problems with sys.path are mentioned: https://cffi.readthedocs.io/en/latest/embedding.html). However, I'm struggling to understand why does it work in linux but not in mac. In the wrapper.py example below I've made this clear by importing sys and showing the content of sys.path.

Is there a way of telling embedding_init_code to load the environment? Note that I cannot set sys there because it doesn't reach the code defined there.

Here's a minimal (not) working example. I can give some clarifications or prepare something a bit more complicated.

libcffi installed like it says here from brew:
https://cffi.readthedocs.io/en/latest/installation.html#macos-x

I'm using python from brew, but the same is true with python from the system.

I create a virtual environment:

python -m venv myvenv
. myvenv/bin/activate

Then install cffi also like in the link above.

python wrapper.py
clang $(pkg-config python3-embed --cflags) foo.c $(pkg-config python3-embed --libs) -fPIC -shared -o foo.so
clang test.c -o test foo.so
./test

And I get the following error:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named '_cffi_backend'

Note, the fact that the error is about _cffi_backend is a red herring. The problem is that it is getting the packages from outside the virtual environment (and, indeed, if one installs cffi from brew it works... but of course, other packages inside the virtual environment will not be accessible.

A trivial solution is to do
export PYTHONPATH=$(python -c 'import _cffi_backend, pathlib ; print(pathlib.Path(_cffi_backend.__file__).parent)')
before calling the executable test but I'd rather do something more elegant.

`wrapper.py`

import sys
print(f"> sys.executable: {sys.executable}")
print(f"> sys.path: {sys.path}")
import cffi


ffibuilder = cffi.FFI()

ffibuilder.embedding_api("""
    int do_stuff(int, int);
""")

ffibuilder.set_source("foo", "")

ffibuilder.embedding_init_code("""
    import sys
    print(f"> sys.executable: {sys.executable}")
    print(f"> sys.path: {sys.path}")
    from foo import ffi

    @ffi.def_extern()
    def do_stuff(x, y):
        print("adding %d and %d" % (x, y))
        return x + y
""")

ffibuilder.emit_c_code("foo.c")

`test.c`

#include <stdio.h>
int do_stuff(int, int);

int main() {
    do_stuff(1, 2);
    printf("This is only a test\n");
    return 0;
}

(I'm guessing this is going to be due to libffi, but having the possibility of playing with the paths before getting to the initialisation part might solve the issue, at least for me)

[arigo]

I have little experience with Mac and I definitely cannot answer you. Virtual environments are slightly different depending on the platform (and change over time too). Moreover, the design of the embedding of cffi is rather minimal. It's not meant to integrate with any way to provide Python except the most bare one. Basically it assumes it's using the system Python and you can hack sys.path to locate extra modules and packages. Either someone else can help, or else you should ask on a wider platform like stackoverflow.

[thatch]

I think it makes sense to close in favor of python/cpython#66409 - a longstanding issue on the combination of embedded python and virtualenvs.

There's a lot of the startup code that the embedder would have to reproduce, and the normal way it resolves the symlink of Py_SetProgramName does not work for binaries that aren't in myvenv/bin and aren't symlinks.

You asked about "playing with the paths" -- if you want want hacks that hardcode the name of the venv I think your init code can simply sys.path.insert(0, "myenv/lib/python3.10/site-packages") and that might work. I don't know if stdlib is going to come from homebrew python, but you might get lucky.

[nitzmahone]

Closing per above