RuntimeError: Arm Plugin: Nodes from torch_jit are not supported

[YaroslavSchubert]

Hi!
I'm trying to run openvino object_detection python sample with the nanodet-plus model on Raspberry Pi 4 using the arm_cpu plugin and getting the following error:

[ INFO ] OpenVINO Runtime
[ INFO ]        build: 2022.3.0-000-c8e62fddb2b
[ INFO ] Reading model /home/toutaudio/openvino_models/nanodet-plus-m-1.5x-416/FP32/nanodet-plus-m-1.5x-416.xml
[ WARNING ] The parameter "input_size" not found in NanoDet-Plus wrapper, will be omitted
[ INFO ]        Input layer: data, shape: [1, 3, 416, 416], precision: f32, layout: NCHW
[ INFO ]        Output layer: output, shape: [1, 3598, 112], precision: f32, layout: 
Traceback (most recent call last):
  File "object_detection_demo.py", line 304, in <module>
    sys.exit(main() or 0)
  File "object_detection_demo.py", line 189, in main
    detector_pipeline = AsyncPipeline(model)
  File "/opt/intel/openvino_cc/extras/open_model_zoo/demos/common/python/openvino/model_zoo/model_api/pipelines/async_pipeline.py", line 86, in __init__
    self.model.load()
  File "/opt/intel/openvino_cc/extras/open_model_zoo/demos/common/python/openvino/model_zoo/model_api/models/model.py", line 263, in load
    self.model_adapter.load_model()
  File "/opt/intel/openvino_cc/extras/open_model_zoo/demos/common/python/openvino/model_zoo/model_api/adapters/openvino_adapter.py", line 66, in load_model
    self.async_queue = AsyncInferQueue(self.compiled_model, self.max_num_requests)
RuntimeError: Arm Plugin: Nodes from torch_jit are not supported:
        Resize_378 (Interpolate.4)- node input index is out of range;
        Resize_392 (Interpolate.4)- node input index is out of range;

I was able to run the model after converting it using omz_converter --add_mo_arg=--use_legacy_frontend command but the inference speed is very low and varies between 7000-10000 ms. I tried nanodet, nanodet-plus, fp16 and fp32 - same problem.
The detection accuracy is good, though.

Is this a desired behaviour in terms of speed for Nanodet model on ARM CPU? Is there a way to optimize it to run at <100ms ?

Thanks in advance!

---

System info:
OS: Raspbian Buster armv7l
Device: Raspberry Pi4
Openvino: 2022.3.0-000-c8e62fddb2b (cross-compiled via docker)
Openvino dev tools (omz_converted, omz_downloader) : 2022.2.0.dev20220829

[alvoron]

Hi @YaroslavSchubert

Thank you for your feedback. Most probably, the significant latency is a result of executing non-optimized reference OpenVINO code. The list of layers supported by ARM plugin is mentioned here: https://github.com/openvinotoolkit/openvino_contrib/wiki/ARM-plugin-operation-set-specification
Some layers are marked Supported** (with two asterisks) it means this layers are calculated by non-optimized reference code.

Could you please confirm that one or more layers in your network are not optimized?

[YaroslavSchubert]

Thanks for your reply.
I will get back to you with an answer.

[alvoron]

@YaroslavSchubert did you have a chance to check what operations are used in the model?
Also, since September a lot of enhancements were merged to master, could you try to run the model on the latest master?