Discard round-to-even code?
ledvinap opened this issue · 1 comments
Round-to-even in %f
is not working correctly, maybe it would be best to simply discard it completely? Opinions?
unsigned long whole = (unsigned long)value;
double tmp = (value - whole) * pow10[prec];
unsigned long frac = (unsigned long)tmp;
diff = tmp - frac;
The problem is that tmp
value is multiplied by pow10
.
With %.1f
, 0.95:
0.95 is represented as 0.94999999999999996 in binary. This should be rounded down.
But (0.95 - 0) * 10
is 9.5, exactly representable in binary and tie-decision code takes place, rounding to even and outputting "1.0"
.
It may be possible to fix this double-rounding (at least in some cases), but IMO it's not worth the complexity. It may be simpler to just remove the tie-handling code and live with it
One edge case is %.0f
- current rounding computation is correct and .5 is exactly representable in binary, so rounding should apply.