waylonflinn/weblas

Unit Test Float Encode Exhaustively

waylonflinn opened this issue · 2 comments

The floating point encode step is a tricky bit to get right. Issue #10 shows that numerical stability issues can make a big difference in some edge cases.

Encoding each unique floating point value as an element of a 4098 square matrix should require only a few hundred of these matrices. This should be doable in a reasonable amount of time.

References

What is the current decimal precision? Is it better, for example than: https://github.com/gpujs/gpu.js/blob/develop/test/src/internal/precision.js#L7?