rationale considering in using floor or round
Opened this issue · 1 comments
CaoZhongZ commented
❓ What is the rationale behind floor or round
I see softmax and polynomial use floor but other places use round. What is the consideration?
kssteven418 commented
For quantization operations (e.g., QuantLinear), we normally use a round-to-nearest policy instead of floor as they are more sensitive to rounding errors. Rounding produces less error than floor in general.
When computing polynomial (e.g., softmax), I simply used floor operation as it is less sensitive to rounding error. However, you can use rounding as well.