Unit Test Float Encode Exhaustively
waylonflinn opened this issue · 2 comments
waylonflinn commented
The floating point encode step is a tricky bit to get right. Issue #10 shows that numerical stability issues can make a big difference in some edge cases.
Encoding each unique floating point value as an element of a 4098 square matrix should require only a few hundred of these matrices. This should be doable in a reasonable amount of time.
References
waylonflinn commented
robertleeplummerjr commented
What is the current decimal precision? Is it better, for example than: https://github.com/gpujs/gpu.js/blob/develop/test/src/internal/precision.js#L7?