foss-for-synopsys-dwc-arc-processors/embarc_mli

do you have plan to optimize leaky_relu and tanh op for tflm?

Opened this issue · 16 comments

nyadla-sys commented

We are using himax board to run our custom model that uses leaky_relu and tanh ops on arc processor and currently we are running on tflm with C reference code and it takes lot of cycles to run inference, so could you please accelerate these ops on TFLM.

nyadla-sys commented

We are in the process of enabling the below kernel for TFLM on HiMax WE1 board as one of our NN models uses the Tanh kernel heavily.
So we like to accelerate Tanh on ARC processors.
mli_status mli_krn_tanh_fx8(const mli_tensor * in, mli_tensor * out);
Do you plan to support this kernel for TFLM? If not please give us guidance to accelerate this kernel on TFLM for ARC EM9 processor(Himax WE1 board).

nyadla-sys commented

@mfarag13, @JaccovG, @Hakim7267 Please comment on this..
Actually I have ported mli_krn_leaky_relu_fx8(const mli_tensor * in, mli_tensor * slope_coeffs,mli_tensor * out) to tflm,however it is not outputting expected result.
if you are open to provide the feedback, I can share the patch which does leaky_relu acceleration on ARC processor on TFLM.

JaccovG commented

Hi,
Feel free to share the patch, I can review it.

nyadla-sys commented

@JaccovG
Please refer the attached patch along with changed leaky_relu files
https://drive.google.com/drive/folders/1lzRuglfxr4QXm_H2NRj3bwYyux_ZL42t?usp=sharing

nyadla-sys commented

Here is output of one test from tflm leaky_relu_test.cc

Entering to prepare of is_mli_applicable
params->alpha 1.0*2^-1

fixed 8: 0x40

tensor->data.int8:0x40

Exiting to prepare of is_mli_applicable
Inside LeakyReluEval params->alpha:1.0*2^-1
Converted to Q7 fixed point tensor->data.int8:0x40

Entering EvalMLI

params->alpha:1.0*2^-1
mli tensor of slope coeffs Q7 fixed point :0x40
res:0

Exiting EvalMLI

expected_data[i] (1.02^0) near output_data[i] (1.59999932^1) failed at examples/kernel_add_test/add_test.cc:103
expected_data[i] (1.49999992^1) near output_data[i] (1.14999942^3) failed at examples/kernel_add_test/add_test.cc:103
expected_data[i] (1.02^0) near output_data[i] (1.59999932^-3) failed at examples/kernel_add_test/add_test.cc:103
expected_data[i] (-1.02^-1) near output_data[i] (1.59999932^1) failed at examples/kernel_add_test/add_test.cc:103
expected_data[i] (-1.02^0) near output_data[i] (1.02^-127) failed at examples/kernel_add_test/add_test.cc:103
Testing QuantizedActivationsOpTestLeakyReluInt8_2

nyadla-sys commented

@JaccovG Could you please review patch and let us know your inputs

nyadla-sys commented

@JaccovG Gentle reminder.!

JaccovG commented

I'm not able to access the google drive. could you share it as a github commit? or as a PR?

nyadla-sys commented

@JaccovG Gentle reminder.!

nyadla-sys commented

@JaccovG Gentle reminder.!

JaccovG commented

sorry for my late reply, I was very busy.
I had a look at your code, and when you set the slope tensor, you force the exponent to 7.
https://github.com/usefulsensors/for_synopsys_review/blob/main/leaky_relu.cc#L91
I don't know the reason for setting it to 7, but maybe the problem is related to how the slope tensor is constructed.

nyadla-sys commented

@JaccovG I have created q7 format for slope tensor,if this is something wrong could you please suggest.
What is the correct implementation for this

JaccovG commented

I couldn't quickly find how you did the conversion. it is fine to use q7 format as long as you shift the mantissa to match the exponent of 7. So what you need to check is if the slope value is correctly converted to the fixedpoint value.