cmpute/dashu

incorrect precision in `RBig::to_float`

Shoeboxam opened this issue · 3 comments

The following assertion panics:

assert_eq!(RBig::ZERO.to_float::<Up, 2>(20).value().precision(), 20);

In Repr::to_float there is an edge-case that returns FBig::ZERO (with precision zero) without updating the precision.

EDIT: This seems to be a common issue throughout the library: precision is not preserved when any edge-cases are encountered (see ln_internal, exp, powf). This makes me think the behavior might be intentional? If so, feel free to disregard.

cmpute commented

Sorry that I missed this ticket. I think this is a bug, whenever a precision is explicitly defined in the arguments, the output should always have that precision. However, the value FBig::ZERO is intentionally designed to have a precision zero (unlimited precision), because it's a exact value and it's exposed as a common constant.

I haven't checked this principle through out the project, but I will try to make it consistent when I complete the guide of the project, and these corner cases will be fixed! Thanks for your report!

The fix has been pushed. But I think it's too small a change to create a new version. Please kindly wait for the next release :)

Agreed, and there's easy workarounds. Thanks!