m-ou-se/inline-python

Error when importing python packages

Closed this issue · 10 comments

PosoSAgapo commented

The python enviroment that I use in my rust package is anaconda python, however, error happened when I was tring to import pandas and many other packages

/opt/anaconda3/lib/python3.7/site-packages/numpy/__init__.py:140: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
  from . import _distributor_init
Traceback (most recent call last):
  File "src/main.rs", line 5, in <module>
    import pandas
  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 17, in <module>
    "Unable to import required dependencies:\n" + "\n".join(missing_dependencies)
ImportError: Unable to import required dependencies:
numpy:

It shows I did not install numpy so that the inline-python could not import pandas, but the numpy is absolutely installed in my python directory.But the default package like sys works fine, then I use import sys to show the import directory of the python that I use in rust, the import directory is just the same as my anaconda python, which both are the same:

sys.path
['', '/opt/anaconda3/lib/python37.zip', '/opt/anaconda3/lib/python3.7', '/opt/anaconda3/lib/python3.7/lib-dynload', '/opt/anaconda3/lib/python3.7/site-packages', '/opt/anaconda3/lib/python3.7/site-packages/aeosa']

What could have possible caused this problem?

de-vri-es commented

The error message you're showing starts with a warning from numpy, so python can find it.

My guess is that while numpy calls it a warning, pandas or maybe something else is treating it like an error. Either way, it sounds like you should install the mkl-service package.

PosoSAgapo commented

The error message you're showing starts with a warning from numpy, so python can find it.

My guess is that while numpy calls it a warning, pandas or maybe something else is treating it like an error. Either way, it sounds like you should install the mkl-service package.

I checked mkl-service and it was installed, the problem is that running the same python in terminal is working normally, but error only happens in the rust project,which is weird, because they are suppose to be the same python installed through anaconda. I also checked the sys.path to make sure the package import directory is same in both rust project and terminal python.

de-vri-es commented

Try to import the mkl-service package directly and see what happens. Maybe numpy is hiding the error that occurs when importing it.

PosoSAgapo commented

I tried directly import mkl, the error is different from import pandas, which shows the below error information:

Traceback (most recent call last):
  File "src/main.rs", line 12, in <module>
    import mkl
  File "/opt/anaconda3/lib/python3.7/site-packages/mkl/__init__.py", line 48, in <module>
    with RTLD_for_MKL():
  File "/opt/anaconda3/lib/python3.7/site-packages/mkl/__init__.py", line 33, in __enter__
    import ctypes
  File "/opt/anaconda3/lib/python3.7/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: dlopen(/opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Symbol not found: _PyByteArray_Type
  Referenced from: /opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
  Expected in: flat namespace
 in /opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
thread 'main' panicked at 'python!{...} failed to execute', /Users/chenbowen/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd/macros.rs:13:23
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It directly shows a import error and did not find the _PyByteArray_Type, but the same python works fine when importing the same package

m-ou-se commented

Oh, interesting. Looks like some dynamic linker stuff is going wrong on Mac. I don't have a Mac to test things on though. I assume import ctypes gives the same error for you?

fn main() {
    python! {
        import ctypes;
    }
}
m-ou-se commented

What's the output of otool -L target/debug/yourprogram?

m-ou-se commented

These look related:

It looks like this is not an issue with iniline-python, but with PyO3 or your Python installation. To make sure, you can check if importing the module directly with pyo3 works:

fn main() {
	pyo3::Python::acquire_gil().python().import("ctypes").unwrap();
}

If that also fails, please open an issue with PyO3.

PosoSAgapo commented

Oh, interesting. Looks like some dynamic linker stuff is going wrong on Mac. I don't have a Mac to test things on though. I assume import ctypes gives the same error for you?

fn main() {
    python! {
        import ctypes;
    }
}

Importing Ctype also cuasing an error information:

 File "src/main.rs", line 12, in <module>
    import ctypes;
  File "/opt/anaconda3/lib/python3.7/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: dlopen(/opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so, 2): Symbol not found: _PyByteArray_Type
  Referenced from: /opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
  Expected in: flat namespace
 in /opt/anaconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so
thread 'main' panicked at 'python!{...} failed to execute', /Users/chenbowen/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd/macros.rs:13:23
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And directly importing it from Pyo3 also casuing a problem:

 thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { type: Py(0x10a947920, PhantomData) }', src/main.rs:14:59
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It seems this might be a problem of PyO3 or my Python installation, but why the python that runs in the rust project does not behave the same way like the terminal python? They are the same python that I checked using sys.version and also the import directory is also the same which I checked using sys.path

PosoSAgapo commented

otool -L target/debug/yourprogram

Running this command, the output is like below:

target/debug/ruststudy:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)

m-ou-se commented

And directly importing it from Pyo3 also casuing a problem

In that case, please open an issue with the PyO3 project. If it isn't working in PyO3, there's nothing the inline-python crate can do. (Make sure to add a link to this issue for context. :) )