PACK op not supported for int8 models?

Question

PACK op not supported for int8 models?

dmartinez-quercus opened this issue 3 years ago · 4 comments

Hello there,

I'm using the well-known SSDMobilenetv2 quantized model with the tflite-vx-delegate on the iMX8M+ NPU. I've noticed the inference process is fragmented into multiple parts due to some nodes of an unsupported operation located at the "middle of the graph". It seems the PACK operation is being delegated to CPU; however, I can see the PACK/STACK operation is implemented in all involved libs: tflite-vx-delegate, TIM-VX, and OVXLIB.

I've checked the source code of this repository and in the "op_map.cc" file at L1766 there is a "IsOpSupported" method for the "PackMapper" struct which returns false if the input tensor data type is INT8.

Does this means that the tflite-vx-delegate does not support the PACK operation for INT8 quantized models so these nodes are delegated and executed on CPU?

Thank you for your time!

Answer 1 · 2022-02-23T04:10:50.000Z

@dmartinez-quercus,

Thanks for feedback and check the code.

You can just enable INT8 support fot SSDMobileNetV2. It can support in this model. We found a corner case which may trigger internal bug, we would like to enable it after the fix available.

Thanks

Answer 2 · 2022-02-23T08:41:31.000Z

@dmartinez-quercus,

Thanks for feedback and check the code.

You can just enable INT8 support fot SSDMobileNetV2. It can support in this model. We found a corner case which may trigger internal bug, we would like to enable it after the fix available.

Thanks

Thanks for the fast reply.

How I can enable INT8 support for the SSDMobileNetV2?

The model was created and trained by the Tensorflow 2 Object Detection API and converted (and quantized) to Tensorflow Lite.
Even having the PACK operations delegated back to CPU, the results are not consistent (it does not detect anything). If I use the NNAPI delegate, the same model runs well even though it has more nodes delegated to CPU than using the vx-delegate.

I suspect the cause of this problem might be related to this unsupported operation issue but I'm aware it can be due to something else.

Answer 3 · 2022-02-25T05:17:27.000Z

@dmartinez-quercus

Please try with latest vx-delegate. Thanks

Answer 4 · 2022-03-19T23:38:52.000Z

@dmartinez-quercus

Please try with latest vx-delegate. Thanks

Hi! Sorry for the very late reply.

I tried this fix and it seems now it's working and the output is consistent.

However, now I get a bunch of the following warnings:
W [HandleLayoutInfer:257]Op 18: default layout inference pass.

Anyway, this fix seems to have solved this problem. Thank you!! :D