paupino/rust-decimal

How to serialize Decimal and still keep the results in order (bitwise comparison)

KKould opened this issue · 3 comments

KKould commented

I will store Decimal as a Key in RocksDB, but how can I keep Decimal in order after serialization and arrange Decimal in a more reasonable manner?

paupino commented

Hi @KKould,

I'm not sure I quite understand the question here - but admittedly, I'm unfamiliar with RocksDB. If this is about key clustering, this may be more suitable to ask on the RocksDB project.

That said - I'll try to provide a couple of answers:

  • Rust Decimal has a c-repr feature if you wanted to align structure across 128 bits consistently. This would allow you to perform memory translation if needed.
  • There is a serialize function to consistently serialize into a collection of bytes. This is in a well-defined order.
  • You could also use something like Borsh to perform binary serialization.

I'm not sure if this helps at all. If not, if you could please provide some further context it may help direct me a bit more.

KKould commented

Hi @paupino

Implement Memory Comparable: After serialization, it can still be sorted (same order as before serialization)
e.g. TIDB: https://github.com/pingcap/tidb/blob/48f6b35d442289721af6fd5753b4b9c151e1d384/util/codec/decimal.go#L25C22-L25C22

I tried serialize but it's not what I want

paupino commented

Another thing you could try is to implement this manually. For example:

Then recombine them for your requirements. The code you've linked seems to write precision, scale then mantissa. You could emulate this by calculating the precision (i.e. length of the mantissa), followed by scale, followed by mantissa. Perhaps that would do the trick?

i.e. in pseudo like code

let mantissa = number.mantissa();
let scale = number.scale();
let precision = mantissa.ilog10() + 1;
// This is not space efficient
let mut bytes = Vec::new();
bytes.extend(precision.to_be_bytes());
bytes.extend(scale.to_be_bytes());
bytes.extend(mantissa.to_be_bytes());
  • 👍1