google/gemmlowp

is D in your fomation (2) calculated by this?

mittkongg opened this issue · 0 comments

max = scale * (255 - zero_point);
min = scale * (0 - zero_point);

and this zero_point and scale both are real number , so is not calculated like what i do ?