decimalExpansion gives out unexpected values
chokokatana opened this issue · 3 comments
I've tried to use the BDouble for storage but at some point I need to use decimalExpansion
to format the value to end users, and I was getting really weird values. In a debug session I got the following:
(lldb) po percent
▿ 7450921770987/7450936270987
- sign : false
▿ numerator : 1 element
- 0 : 7450921770987
▿ denominator : 1 element
- 0 : 7450936270987
- magnitude : 0.0
- _precision : 4
(lldb) po percent.decimalDescription
"0.99910"
(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 2)
"0.910"
(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 0)
"0.0"
(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 7)
"0.9999981"
I'm pretty certain a value nearly 1 should not round to 0 when displayed to end users. Is decimalExpansion
not the function I'm expecting and should I be using something else? I had expected the value to round to either 1.0
or 0.9
.
Can you set the "rounded" parameter to false and check again? I think the problem might have something to do with rounding at a given precision level.
But still that doesn’t make sense why for example the case with 2 doesn’t give at least two 9's
Okay I got it, in the case of precision=2, it computes 0.999 so since the third 9 is >= 5, the second 9 is rounded up, but there is a bug so it is replaced with 10 and the number becomes 0.910
Could be solved easily by checking the last digit of the derived BInt before converting to a string, if .divmod(10).remainder >= 5 then take .divmod(10).quotient + 1, else .divmod(10).quotient. Only after that take the Bint.description and prune 0's from the end if needed (so if 999 is rounded to 1000, then 1.000 has to remove the last zero