f64 precision problem
Closed this issue · 1 comments
whsqjss commented
rekka commented
This is a limitation of how computers handle floating point numbers so there is not much that can be done about this if we want to use the fast hardware floating point operations. See for example Python documentation: Floating Point Arithmetic: Issues and Limitations
You can try the same expression in Rust directly:
fn main() {
let x = (2.27 - 2.) / 3.;
println!("{x}");
let x = 0.3 * 3.;
println!("{x}");
}
This prints
0.09000000000000001
0.8999999999999999
Hope this helps.