intel/openvino-rs

crate::core::Core::read_model_from_buffer example?

BulbaWarrior opened this issue · 4 comments

BulbaWarrior commented

It is not very obvious how to create a model from memory.
Specifically, I couldn't figure out how to make a Tensor from a Vec<u8> read from a file.
The most obvious way to achieve that is Tensor::from_ptr, but it is only pub(crate)

abrown commented

Thanks for the report; take a look at #137 and let me know if that explains things well-enough.

BulbaWarrior commented

Thanks for the swift reply! The example is great but still does not cover my particular use case. My goal is to acquire a Model object from files in OpenVino IR format without providing file paths directly.
For example this could be useful if I download model files over network and don't want to store them on the file system.
So in my mind the example looks like this:

use openvino::Core;
use std::fs;

fn main() -> anyhow::Result<()> {
    let mut core = Core::new()?;
    // imagine graph and weights are instead downloaded over network
    let graph = fs::read("model.xml")?;
    let weights = fs::read("model.bin")?;

    let weights_tensor = todo!(); // change me
    let model = core.read_model_from_buffer(&graph, Some(&weights_tensor))?;
    Ok(())
}

I guess your example works, but then how do I get the shape of the weights tensor, should I parse the graph file?

abrown commented

Ah, I see the missing link. What I put up in #137 is what you need in order to fill out the todo!() in your latest snippet but I didn't consider the read_model_from_buffer part which was your original question! Essentially, create a tensor from the weights bytes using the U8 and a 1 x <# bytes> shape and copy in the bytes like I showed in #137. Then you can pass that tensor to core_read_model_from_buffer. Here's an example of how this is used in Wasmtime: https://github.com/bytecodealliance/wasmtime/blob/8ea2590/crates/wasi-nn/src/backend/openvino.rs#L38-L41.

I'll merge #137 since that is helpful regardless but maybe we should use your snippet as a doc example on read_model_from_buffer. The problem is we would need a valid model but, hm, maybe we can just ignore the error... Thoughts?

  • 👍1
BulbaWarrior commented

Okay, so I wrote a small example test for reading a model from buffers: #138

But I think I stumbled into an upstream bug in the newer versions on openvino: passing ElementType:U8 does not allocate enough space for the tensor (at lest it seems so since the size coming from ov_tensor_get_byte_size is definitely smaller then expected (91447851 bytes vs expected 243860936 for alexnet). The bug happens somewhere between 2024.0.0 and 2024.2.0, but I couldn't pin it down for now

But it also shows that having an honest-to-god-not-failing test is valuable here, so this is where I think it belongs

  • 👍1