LLNL/unum

Errors in some calculations

Closed this issue · 1 comments

Hi,
I am noticing some peculiarities in some of the calculations. For example:
I expect this:

r-237-121-25-172:lab5 hugh$ ./unumcalc 1 div "[0,2]"
   Result of 1 div [0,2] is NaN
r-237-121-25-172:lab5 hugh$ ./unumcalc 1 div "(0,2]"
   Result of 1 div (0,2] is [0.5,Inf)

But I am surprised by this:

r-237-121-25-172:lab5 hugh$ ./unumcalc 1 div "[0,1]"
   Result of 1 div [0,1] is NaN
r-237-121-25-172:lab5 hugh$ ./unumcalc 1 div "(0,1]"
   Result of 1 div (0,1] is NaN

It appears that if the dividend is the same value as the upper bound of the divisor, then the result is wrong:

r-237-121-25-172:lab5 hugh$ ./unumcalc 2 div "(0,2]"
   Result of 2 div (0,2] is NaN
r-237-121-25-172:lab5 hugh$ ./unumcalc 3 div "(0,3]"
   Result of 3 div (0,3] is NaN
r-237-121-25-172:lab5 hugh$ ./unumcalc 4 div "(0,4]"
   Result of 4 div (0,4] is NaN

Anyway - has anyone else noticed this behaviour?
Cheers Hugh

The root cause of this problem is the same as that reported by @milthorpe as pull request #1. The result of the division in your examples are correct in the g-layer, but were incorrectly flagged as NaNs when converted to the u-layer by g2u. A test case from your example has been added to tbasic.c.