Please warn if no rust is exported in a mixed rust/python module
gilescope opened this issue ยท 6 comments
When I create a mixed rust and python project, it seems one needs to re-export the rust into the python.
It would be great if maturin warned if nothing was imported from the rust module - i.e. the user thought it happened automagically.
@gilescope I am having a similar problem. This took me some messing around.
In the end the key was inside the init.py in the "myproj" python src dir:
# re-export rust myproj module at this level
from .myproj import *
# export vanilla_python.py functions as vanilla_python module
from . import vanilla_python
Now I can import myproj.rust_ffi and also myproj.vanilla_python on the same package level.
So while this is letting me execute the full path to functions, I have an issue with a nested module.
#[pymodule]
fn myproj(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pymodule!(message))?;
Ok(())
}
#[pymodule]
fn submod(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pyfunction!(run_class_method_message))?;
Ok(())
}
With this I can do:
import myproj
myproj.submod.run_class_method_message()
And I can do:
from myproj import submod
But I can't do:
from myproj.submod import run_class_method_message
I get the error:
ModuleNotFoundError: No module named 'myproj.submod'
Am I doing something wrong?
But I can't do:
from myproj.submod import run_class_method_message
I get the error:
ModuleNotFoundError: No module named 'myproj.submod'
Am I doing something wrong?
no, that's a limitation in how CPython loads extension modules.
Okay, so I have actually found a way to fix this using some python duct tape.
The fix below allows you to import normal vanilla python libs like:
from myproj.vanilla import abc
As well as rust modules at depth like:
from myproj.suba.subc import xyz
Your directory structure:
./
โโโ Cargo.toml
โโโ myproj. <-- this is your mixed python src
โ โโโ init.py
โ โโโ import_fixer.py
โ โโโ suba
โ โ โโโ init.py
โ โ โโโ subc
โ โ โโโ init.py
โ โโโ subb
โ โ โโโ init.py
โ โโโ vanilla.py
โโโ src
Your rust modules:
#[pymodule]
fn myproj(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pymodule!(suba))?;
m.add_wrapped(wrap_pymodule!(subb))?;
Ok(())
}
#[pymodule]
fn suba(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pyfunction!(geh))?;
m.add_wrapped(wrap_pymodule!(subc))?;
Ok(())
}
#[pymodule]
fn subb(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pyfunction!(def))?;
Ok(())
}
#[pymodule]
fn subc(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pyfunction!(xyz))?;
Ok(())
}
Inside myproj/init.py:
# here you can import the normal kind of vanilla python __init__ that you want
from . import vanilla.py
For each submodule from rust you want, add a directory and a init.py
In each init.py put:
import importlib
# find and run the import fixer
package_name = __name__.split(".")[0]
import_fixer = importlib.import_module(".import_fixer", package=package_name)
import_fixer.fix_imports(locals(), __file__)
Finally inside import_fixer.py put:
# -*- coding: utf-8 -*-
# Author: github.com/madhavajay
"""Fixes pyo3 mixed modules for import in python"""
import importlib
import os
from typing import Dict, Any, List
# gets the name of the top level module / package
package_name = __name__.split(".")[0] # myproj
# convert the subdirs from "package_name" into a list of sub module names
def get_module_name_from_init_path(path: str) -> List[str]:
_, path_and_file = os.path.splitdrive(os.path.dirname(path))
module_path = path_and_file.split(package_name)[-1]
parts = module_path.split(os.path.sep)[1:]
return parts
# step through the main base module from rust at myproj.myproj and unpack each level
def unpack_module_from_parts(module: Any, module_parts: List[str]) -> Any:
for part in module_parts:
module = getattr(module, part)
return module
# take the local scope of the caller and populate it with the correct properties
def fix_imports(lcl: Dict[str, Any], init_file_path: str, debug: bool = False) -> None:
# rust library is available as package_name.package_name
import_string = f".{package_name}"
base_module = importlib.import_module(import_string, package=package_name)
module_parts = get_module_name_from_init_path(init_file_path)
submodule = unpack_module_from_parts(base_module, module_parts)
if debug:
module_path = ".".join(module_parts)
print(f"Parsed module_name: {module_path} from: {init_file_path}")
# re-export functions
keys = ["builtin_function_or_method", "module"]
for k in dir(submodule):
if type(getattr(submodule, k)).__name__ in keys:
if debug:
print(f"Loading: {submodule}.{k}")
lcl[k] = getattr(submodule, k)
Which begs the question, could this be autogenerated and included in pyo3?
When I create a mixed rust and python project, it seems one needs to re-export the rust into the python.
That is not necessary. In the pyo3_mixed example you can e.g. do from pyo3_mixed import pyo3_mixed; pyo3_mixed.get_21()
. However as @programmerjake said, cpython can't import from a submodule of a native module directly.
@konstin, what are your thoughts on my solution above?
It provides the ability to mix both vanilla python and native modules / functions on the same import syntax and path with minimal effort. PyO3 could have a build flag which prepends this to any "/module/subdir/init.py" files inside the mixed python source providing these re-exports in a way that the user thinks logically and allowing any existing custom code in init to overwrite the locals() keys if desired.