rekka/meval-rs

f64 precision problem

Closed this issue · 1 comments

(2.27 - 2) / 3 should be 0.09, but got 0.09000000000000000001

Is there a solution to this problem?
Looking forward to your reply. Thanks
err

This is a limitation of how computers handle floating point numbers so there is not much that can be done about this if we want to use the fast hardware floating point operations. See for example Python documentation: Floating Point Arithmetic: Issues and Limitations

You can try the same expression in Rust directly:

fn main() {
    let x = (2.27 - 2.) / 3.;
    println!("{x}");
    let x = 0.3 * 3.;
    println!("{x}");
}

This prints

0.09000000000000001
0.8999999999999999

Hope this helps.