wader/fq

[feature] add decimal floating-point number support

matthewdale opened this issue · 2 comments

matthewdale commented

The BSON and Avro formats support encoding decimal floating-point numbers which cannot be exactly represented by a float64. Currently, neither the bson or avro decoder packages actually support decoding the respective decimal number value, only displaying the values as binary. There are two main things that stand in the way of decoding BSON and Avro decimal number values:

  • The decode.D type doesn't support decoding decimal floating-point numbers.
  • The wader/gojq library doesn't support arithmetic on decimal floating-point numbers.

Add support for decoding various decimal floating-point binary formats. Consider using the shopspring/decimal library for the decoded value representation because it can represent almost any decimal value (except NaN, +Inf, -Inf).

wader commented

I wonder if we should ask the gojq author what he thinks of decimal support. He has already extended from original jq to support arbitrarily precision so maybe he has some ideas. Would be nice to not have too much difference compared to upstream.

btw any idea if big.Rat from standard library would be ok also?

matthewdale commented

Good idea, I opened an issue on gojq requesting support for arbitrary precision non-integer decimal values: itchyny/gojq#216

I believe big.Rat can represent all numbers that shopspring/decimal can represent and seems to support the required arithmetic operations (except modulus, which would probably be accomplished by truncating the big.Rat to a big.Int and using that to calculate modulus). The main difficulty with big.Rat in the context of fq is that to get a decimal-with-radix representation (i.e. "123.4" instead of "1234/10") you use FloatString, which requires you to specify (and keep track of) the required number of digits of precision yourself. It's worth an experiment to validate if it's possible.