Is it possible to pass a tokenizer from Python into Rust?
albertsgarde opened this issue · 1 comments
I am implementing a part of my Python project in Rust. I need to use a tokenizer deep in the Rust code, but I want the Python code to decide what tokenizer to use. I would therefore like to pass a tokenizer from Python to Rust. Since the Python wrapper is supposed to be thin, I have hope that this is possible. If it is not, I'll probably just use Pyo3 to call Python from Rust, but that would both be uglier and feel silly to call Python code that only calls right back into Rust.
What I have tried looks something like this:
use tokenizers::Tokenizer;
#[pyfunction]
pub fn foo(tokenizer: &Tokenizer) {
baz(tokenizer)
}
but Pyo3 gives me the error "the trait bound &tokenizers::Tokenizer: pyo3::PyClass
is not satisfied". Looking at the source, this makes sense since it is the PyTokenizer
struct in the crate tokenizers-python
that wraps Tokenizer
and is turned into a Python class. If I could access the PyTokenizer
struct in Rust I might therefore be able to fix the problem, but since tokenizers-python
is not available on crates.io this is difficult.
Is there some way to pass a tokenizer from Python to Rust?
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.