Unexpectedly bad precision

[gunnarmein]

Hello Martin,

I ran this: nerdamer("0.0000001885 * 1000000000").text()
and the output is '188.500000074053571458'

That's surprisingly bad. Any idea whether this is expected? Thanks for looking at it!

[gunnarmein]

Here is where this originated, if you are curious:

nerdamer("1.885e-7*1e9") -> massive rounding errors

math.js will do a good job in this form, terrible (like nerdamer) in the form above.

[gunnarmein]

I think the real issue here is that the number is converted from scientific notation into a text decimal before finally making it into a fraction. A better approach for scientific notation numbers with negative exponents would be to convert them into a fraction right away, or even to scale the mantissa so that neither numerator nor denominator have decimals. I will work around this problem by doing my own pre-encoding of scientific numbers as fractions, but I would recommend to fix the root cause.

[gunnarmein]

Correcting myself: That doesn't just go for scientific notation, but really all decimals, as the simple imprecision example above shows.

[jiggzson]

@gunnarmein, I'll look into this. These precision issues have been a pain and part of it is self-inflicted. The library started off as a simple evaluator with some symbolic capabilities and the decision was made to track numbers as ratios. Hopefully this issue is not related to that but I wouldn't be surprised if it was.

[gunnarmein-ts]

I think internal representation as fractions is not necessarily a bad thing. Converting a number to a text decimal, and then trying to find a close fitting fraction is the problem. I suggest keeping it simple and stupid, and convert scientific notation into a fraction in the most straight-forward way. If it is of any help, below is what I am using on top of nerdamer to feed it numbers:

numToFrac(n: string):string {
    let NUMBER = /(([-+]?[0-9]+)(\.[0-9]+)?)([eE]([-+]?[0-9]+))?/g
    return n.replace(
        NUMBER,
        (match: string, p1: string, p2: string, p3: string, p4: string, p5: string) => {
            let int: string = p2;
            let digits: string = p3 ? p3.substr(1) : "";
            let numdigits: number = p3 ? p3.length - 1 : 0;
            let exp: number = p5 ? Number(p5) : 0;
            exp -= numdigits;

            if (exp > 0) {
                return "(" + int + digits + "*" + ("1".padEnd(exp + 1, "0")) + ")";
            } else if (exp < 0) {
                return "(" + int + digits + "/" + ("1".padEnd(-exp + 1, "0")) + ")";
            }
            return int + digits;
        });
}